Experimental Design and Some Threats to Experimental Validity: A Primer

Experimental designs are distinguished as the best method to respond to questions involving causality. The purpose of the present paper is to explicate the logic of experimental design and why it is so vital to questions that demand causal conclusions. In addition, types of internal and external validity threats are discussed. To emphasize the current interest in experimental designs, EvidenceBased Practices (EBP) in medicine, psychology and education are highlighted. Finally, cautionary statements regarding experimental designs are elucidated with examples from the literature. Experimental Design 3 The No Child Left Behind Act (NCLB) demands “scientifically based research” as the basis for awarding many grants in education (2001). Specifically, the 107th Congress (2001) delineated scientifically-based research as that which “is evaluated using experimental or quasi-experimental designs”. Recognizing the increased interest and demand for scientifically-based research in education policy and practice, the National Research Council released the publication, Scientific Research in Education (Shavelson & Towne, 2002) a year after the implementation of NCLB. Almost $5 billion have been channeled to programs that provide scientifically-based evidence of effective instruction, such as the Reading First Program (U. S. Department of Education, 2007). With multiple methods available to education researchers, why does the U. S. government show partiality to one particular method? The purpose of the present paper is to explicate the logic of experimental design and why it is so vital to questions that demand causal conclusions. In addition, types of internal and external validity threats are discussed. To emphasize the current interest in experimental designs, Evidence-Based Practices (EBP) in medicine, psychology and education are highlighted. Finally, cautionary statements regarding experimental designs are elucidated with examples from the literature. Experimental Design An experiment is “that portion of research in which variables are manipulated and their effects upon other variables observed” (Campbell & Stanley, 1963, p. 171). Or stated another way, experiments are concerned with an independent variable (IV) that causes or predicts the outcome of the Experimental Design 4 dependent variable (DV). Ideally, all other variables are eliminated, controlled or distributed in such a way that a conclusion that the IV caused the DV is validly justified. Figure 1. Diagram of an experiment. In Figure 1 above you can see that there are two groups. One group receives some sort of manipulation that is thought (theoretically or from previous research) to have an impact on the DV. This is known as the experimental group because participants in this group receive some type of treatment that is presumed to impact the DV. The other group, which does not receive a treatment or instead receives some type of alternative treatment, provides the result of what would have happened without experimental intervention (manipulation of the IV). So how do you determine whether participants will be in the control group or the experimental group? The answer to this question is one of the characteristics that underlie the strength of true experimental designs. True experiments must have three essential characteristics: random assignment to Outcome measured as DV No manipulation or alternate manipulation of IV (treatment or intervention) Control Group Manipulation of IV (treatment or intervention)