Evaluation of Imputation of Covariates in an Impact Analysis With Regression Adjustment

In an impact analysis using random assignment, researchers often deal with missing values in both the covariates and the outcome variables of regression models. Clearly rigorous methods are needed to impute missing values in the outcome variables to minimize the potential bias in impact assessments. When imputation is applied to covariates of the regression analyses, the effect of imputation is less clear on impact analyses. This paper assesses this effect, using a random assignment evaluation of the Growing America Through Entrepreneurship (GATE) program. Two outcome variables used in the original evaluation are modeled against a set of 10 covariates, a treatment indicator, and variables associated with the site of the evaluation. Impacts are assessed with different types of missingness in the covariates with values imputed using mean imputation and sequential hot deck. Considerable work has been done in the area of statistical inference with missing data. Special attention has been paid to the problem of missing values of the independent variables in regression. Little (1992) reviewed different methods of dealing with missing values in the independent variables, when the outcome variable is not missing. He determined that maximum likelihood methods that are based on both the independent variables and the dependent variable are preferred over other methods, with considerable promise shown for multiple imputation methods as well. Allison (2001) also recommends using likelihood-based methods, though he does indicate the list wise deletion is superior to all imputation methods (but particularly single imputation methods) in some instances. Little and Rubin’s (2003) recommendation is to “impute draws (not means) from the conditional distribution of the missing covariates given the observed covariates and Y. ” They indicate that this will produce consistent estimates of the regression estimates. The results discussed above were obtained by looking at estimates of the mean, variance, and coefficient parameters. In an impact evaluation, treatment status is included as one of the covariates in the regression. Analysts are interested in the difference between the treatment and control groups, often adjusted by controlling for other auxiliary covariates in the regression. The main objective of this study is evaluating what happens when there are missing data in these auxiliary covariates. There may be insufficient resources (or insufficient expertise) to implement sophisticated imputation procedures. The question therefore arises: is it necessary to impute missing values in the auxiliary covariates of a regression when the study is an impact evaluation, where the focus is on the comparison between treatment and control means? If it is necessary, does it matter which method is used? In this paper, we look at the effect of missing data in independent variables in a regression, focusing on the estimated effects of the contrast which compares treatment to control outcomes. We compare list wise deletion with unconditional mean imputation (within treatment group), conditional mean imputation, and unweighted sequential hot deck imputation. For the remainder of this section, we discuss mechanisms that lead to missing data, and options for handling missing data. Section 2 describes the source data and the study from which the data came. The study methodology is described in Section 3. Results are presented in Section 4, and the paper concludes with conclusions and discussion in Section 5.