THE RELATIVE ASSIGNMENT VARIABLE APPROACH TO SELECTION BIAS IN PRETEST-POSTTEST GROUP DESIGNS 1

The purpose of this paper is to explore a potential analytic solution, termed here the "relative assignment variable" approach, to the problem of selection bias in pretest-posttest group designs. The presentation includes a description of the most common pretest-posttest group designs and an explanation of why some of these are susceptible to selection bias problems. In addition there is a discussion of the relative assignment variable approach, and the illustration of this approach through simulations and the analysis of real data. Pretest-Posttest Group Designs and Selection Bias. Pretest-posttest group designs have in common the existence of an observed pretest, xi, and posttest, Yi, and can be distinguished by the manner in which persons are assigned to condition (e.g., program and comparison groups, program 1 and program 2, etc.) where the assignment is usually represented by the variable z i (i.e., z i = 1 if the research participant is in the program, 0 otherwise). Four such designs are defined here in terms of their assignment strategies. The first of these is the true randomized experiment which is characterized by random assignment to condition. As a result, the groups which are created can, on the average, be assumed equivalent on the pretest (i.e., the expected value, E(xi) , is equal for both groups). Although the true experiment does not require a pretest, the version of the design which is of interest in this paper includes one as a covariate to increase the statistical precision of the program effect estimate. The second of these, the regression-discontinuity design, is characterized by assignment on the basis of a cutoff score on the pretest. That is, all persons scoring on one side of a selected cutoff value are assigned to one condition with the remaining participants assigned to the other. The third of these, the "fuzzy" regression-discontinuity design, occurs when there is misassignment relative to this cutoff value. Here, some persons who should have been assigned by the cutoff to one condition are incorrectly included in the other. Finally, the non-equivalent group design allows for non-equivalence between the groups on the pretest and can occur when the random assignment in a true experiment is not maintained or when individuals or intact groups (e.g., classrooms, agencies, governmental units, etc.) are assigned to condition in a nonrandom manner. In fact, the distinction between the fuzzy regression-discontinuity and nonequivalent group designs is somewhat arbitrary (Reichardt, 1979). As will be shown later, data representative of both designs can be generated for simulation purposes by means of the same general models. Pretest-posttest group designs are commonly chosen techniques for the evaluation of social programs. For information on the use of these methods one can consult Boruch et al (1978) who describe a number of randomized experiments, Cook and Campbell (1979) and Reichardt (1979) who speak of the utility of non-equivalent group designs, and Campbell (1-969) and Trochim (1980) who discuss applications of the regression-discontinuity and fuzzy regression-discontinuity designs. Despite an apparent conceptual similarity beween pretest-posttest group designs in terms of observed xi, Yi and zi, no general small sample analytic strategy is known to the authors. While the regression of Yi on x i and z i (and perhaps polynomials in x i and interactions of x i and zi) will yield unbiased estimates for some of these designs, it cannot be used for all of them, at least in part because of the selection bias problem. The problem of selection bias has been viewed as specification error or omitted-variable bias by Barnow, Cain and Goldberger (1978) who state: "Selectivity bias addresses the question of whether there is some characteristic of the treatment (or control) group that is both associated with receipt of the treatment and associated with the outcome so as to lead to a false attribution of causality regarding treatment and outcome. So stated, selectivity bias is a version of omittedvariable bias, which is commonly analyzed under the rubric of specification error in econometric models. " (p. 4) Selection bias may affect program estimates when a variable related to z i and Yi is not included in the analytic model. The developmen t of a general analytic scheme which can compliment the conceptual similarity of pretest-posttest group designs is seen here then as dependent, at least in part, on an analytic solution to the selection bias problem. The four designs outlined above can be examined for their potential for selection bias. If correctly implemented, the true randomized experiment is free of selection bias because assignment is random and independent of all pre-program measures. Within the context discussed in this paper, Goldberger (1972)has shown that the regression-discontinuity design is free of selection bias as long as assignment is adhered to and the underlying model is linear. This is because assignment is entirely based on a cutoff value on the pretest and any other measures related to assignment must, by definition, be related to the pretest (and will be accounted for in the regression of Yi on x i and z i by the presence of the pretest). In the fuzzy regression-discontinuity design assignment is not based entirely on the pretest alone. In fact, "fuzziness" can be defined as misassignment relative to what the pretest cutoff value would have dictated. As a result, the pretest does not perfectly account for assignment (as in regressiondiscontinuity) and the potential for selection bias exists. This is also the case for the nonequivalent group design. Here, assignment is not based on the pretest but rather on a judgment or determination of the pre-program "equivalence" of the groups. The extent to which the factors which determine the assignment to group affect z i and Yi and are unaccounted for in the analytic model determines the potential for bias. The need for an analytic solution to the selection bias problem in the fuzzy regression-