Multiple Imputation and Higher Education Research

Higher education researchers using survey data often face decisions about handling missing data. Multiple imputation (MI) is considered by many statisticians to be the most appropriate technique for addressing missing data in many circumstances. However, our content analysis of a decade of higher education research literature reveals that the field has yet to make substantial use of this technique despite common employment of quantitative analysis, and that many recommended MI reporting practices are not being followed. We conclude that additional information about the technique and recommended reporting practices may help improve the quality of the research involving missing data. In an attempt to address this issue, we offer an annotated practical example focusing on decision points researchers often face. MI Higher Education 3 Multiple imputation and higher education research Introduction Higher education researchers using large survey datasets frequently face decisions about how to handle missing data, which may influence their results and conclusions. The presence of missing data can potentially affect both the validity and reliability of research findings, and lead to problems with the generalizability of results (see McKnight et al (2007) for a clear elaboration of the potential consequences of missing data). While listwise (or casewise) deletion has often been higher education researchers’ approach for addressing missing data in the past, newer statistical techniques have eclipsed traditional methods of handling missing data in appropriateness for most circumstances (Croninger & Douglas, 2005; Donders, van der Heijden, Stijnen, & Moons, 2006; Schafer & Graham, 2002). In many practical social science research situations, the multiple imputation (MI) technique introduced by Rubin (1987) is considered the “gold standard” for handling missing data (Treiman, 2009, p. 185). The phrase “multiple imputation” refers to a process of proceeding reasonably with statistical analyses given the uncertainty caused by the presence of missing data. Despite entailing a relatively complicated array of steps, the process is approaching the realm of regular use by non-statistically-oriented researchers and is now implemented in most common statistical software packages (e.g. R, SAS, Splus, SPSS, and Stata) in addition to several special packages (e.g. Amelia and SOLAS). Interested researchers should understand that MI is a “state of the art” technique (Schafer & Graham, 2002). This status means that it is constantly being revised and expanding into new areas, and there is uncertainty associated with aspects of the technique still in development or lacking consensus among statistical experts. This means that as researchers, we face subjective MI Higher Education 4 decisions based on our data, and guidance for such questions has been scattered or slowly developed. As a result, while the technique has made significant inroads in certain disciplines such as health (Schafer, 1999), psychology (Graham, 2009), and biostatistics (White, Royston, & Wood, 2011) (also see the list of published research using MI by van Buuren and GroothuisOudshoorn (2011)), it is still being minimally implemented in many fields, including ones that use large data sets (Stuart, Azur, Frangakis, & Leaf, 2009), particularly in education (Peugh & Enders, 2004). The purpose of our paper is to understand the current state of MI in higher education research and based on that understanding, to recommend practical ways to improve its use and reporting grounded in the current state of MI development. We achieve this purpose by: a) generating an aggregated list of recommended practices for reporting based on a review of the methodological MI literature, b) analyzing the content of published higher education research to examine how recommended practices are being used, and c) presenting an example analysis demonstrating recommended MI practices on a higher education topic, while providing a non-statistically-oriented discussion of some of the practical choices to be made when using MI. In doing so, we attempt to facilitate the use of the MI technique by researchers who are interested in following the growing recommendation within the statistical community of using MI and to improve the state of higher education research. Why Is It Okay To Use Multiple Imputation? As a prelude, we feel it important to address a fundamental concern we have heard in discussions with colleagues. Even experienced quantitative researchers in higher education can be highly skeptical of the advisability of using a technique like multiple imputation. It smacks too much of “making up data” for some people’s comfort, and many simply dismiss it since it does not solely use data collected by the researcher (even though it uses information collected by MI Higher Education 5 the researcher, a key distinction to which we will return). Thus, we begin here with a subjective decision point facing experienced skeptics and/or researchers new to MI: whether to continue using listwise deletion (or another traditional method) or learn about MI and implement it if appropriate. If one is not convinced of the merits of the MI approach then what follows in this article will be irrelevant and ineffective in its purpose. There are several well-known problems fundamental to the use of listwise deletion (Acock, 2005; Ludtke, Robitzsch, Trautwein, & Koller, 2007; Peugh & Enders, 2004; Schafer & Graham, 2002). For researchers using large survey datasets, the most relevant of these is that often they will not know whether there are any systematic reasons why particular data are missing, and this means they do not know whether their statistical estimates are biased, potentially causing incorrect conclusions. Thus, the typical assumption that data are missing completely at random (MCAR) and that dropping cases will not affect the validity of results, may be convenient, but may not be correct. MI has been shown to be superior to listwise deletion in almost all circumstances (Allison, 2002), except possibly in cases with very small amounts of missing data and data that is missing completely at random. Even with such data, however, the loss of power that comes with dropping cases may still be problematic, and Graham (2009) argues that MI could always be reasonably used (even with <5% missing data). The bigger question for skeptical researchers is 1 There are numerous clear explanations of the possible types of missing values, traditionally referred to using Rubin’s nomenclature of MCAR, MAR, and MNAR (McKnight et al., 2007; Nakagawa & Freckleton, 2011; Peugh & Enders, 2004; Schafer & Graham, 2002). Essentially, MCAR data are a random sample where any missing data are due to random reasons related neither to observed nor unobserved variables. With data that are MAR, the missing data are either random or related to observed variables, and so will essentially be MCAR once a researcher controls for those observed variables in a statistical analysis. MI Higher Education 6 whether multiple imputation actually offers a reasonable alternative to the known difficulties with listwise deletion and other traditional techniques. Understanding appropriate and inappropriate ways to think about what is happening when data is multiply imputed may help those in doubt. When using MI, you are fundamentally considering the distribution of values in the entire population for certain variables. You are not considering the individual value of a variable for a particular person in a sample. This would indeed be “making up data,” and is expressly not the point of multiple imputation. Schafer (1999) describes MI as “a device for representing missing-data uncertainty. Information is not being invented with MI any more than with. other well accepted likelihood-based methods, which average over a predictive distribution” (p. 8). Recall that each variable in a study has some underlying distribution of expected values within the population as a whole, and the randomly sampled individuals in a survey sample are chosen in order to be representative of that population. Each variable measured has an associated underlying distribution function that represents the probability of particular possible values (with the probabilities across the distribution summing to 1 for a given variable). These variables may be correlated with each other, which means that knowing something about one variable gives you information about another correlate. Multiple imputation is using the fundamental distributional property of the measured variables and their correlations to produce reasonable average estimates of statistical inferences based on the collected information. Thus, it is not necessary to drop cases, thereby losing known correlation information (and the corresponding potentially substantial investment of resources put into collecting the data). Instead, all of the information collected is used to estimate statistical inferences addressing questions you have about relationships shown by the data. MI Higher Education 7 In other words, the general point of MI is not to produce particular values for the data. The point is to produce valid statistical inferences. Using all of the information about your variables actually helps in producing better statistical results than other methods like listwise deletion, given the uncertainty that exists because of the missing data. This is because MI produces smaller standard errors and less bias than other typical methods. Put another way, if you know something about the missing data for a particular variable because of that variable’s correlation with other variables in a model, why would you throw out that information and therefore generate less accurate statistical inferences? It may be the case that analyses using MI and a traditional method like listwise deletion produce similar results. In this situation, it may be fine t