Advances in Missing Data Methods and Implications for Educational Research

The impact of missing data on quantitative educational research has been a great concern to methodologists but less so for many educational researchers. In studies where missing data are present, it is common for educational researchers to use ad hoc methods such as listwise or pairwise deletion to deal with the missing data problem. Yet the recent American Psychological Association Task Force on Statistical Inference warned against the use of these methods for dealing with missing data and encouraged researchers to instead employ alternative methods that take into account the conditions under which missing data occurred. These newer methods have been implemented in statistical software (e.g., SPSS® and SAS®); thus, they are accessible to researchers. In this paper, we promote the use of principled statistical methods for treating missing data that employ singleor multipleimputation of missing values. These methods are illustrated with real-world data sets, and the results are shown to be different from those obtained from listwise deletion method. Statistical assumptions underlying these principled methods are discussed and emphasized. Quality of educational research will be enhanced if (a) researchers explicitly acknowledge missing data problems and the conditions under which they occurred, (b) principled, rather than ad hoc methods, are used to handle missing data, and (c) journal referees and editors incorporate the reporting of appropriate missing data methods in standards used in judging manuscripts for publication. Advances in Missing Data 3 Advances in Missing Data Methods and Implications for Education Research Consider a scenario in which a freshly minted Ph.D. showed off his bound dissertation to a fellow graduate student who was known among her peers to be a “number cruncher. ” As she leafed through the pages of the methods section, she quickly noticed that all missing data were coded zero, instead of being declared as missing. She also saw that all analyses in this dissertation, including factor analysis, multiple regression, discriminant function analyses, etc., were based on all data including the zeros. No one on the dissertation committee caught this mistake. In another scenario, a Ph.D. candidate explaining the results of the data analyses to her dissertation committee noted that the choice of listwise deletion led to the removal of approximately one third of the data. Both the candidate and committee members failed to recognize the possibly devastating effects on inferences due to these missing data. Variations of these scenarios may occur more frequently in educational research than is commonly believed or acknowledged. Missing data in research studies are the rule rather than the exception. For example, various literatures indicate that a 20% attrition rate has been held as a benchmark for studies related to youth, school-based programs, and clinical research (Hall, 1993; Kellam, Rebok, Ialongo, & Mayer, 1994; Mason, 1999). Many reasons contribute to data missing from research projects: subject’s attrition in pretest-posttest or longitudinal designs, equipment failure or unclear instruction or intrusive questions on a survey. Whatever the reason(s) for missing data, their impact on quantitative research has been a great concern to methodologists. In survey research, missing data problems are particularly acute, with substantial amounts of missing data frequently appearing (Little & Rubin, 1987). To combat these problems, investigators spend enormous time, funding, and energy to minimize incomplete data or nonAdvances in Missing Data 4 response among respondents (Mason, 1999). Procedures have been devised by statisticians to mitigate inferential or descriptive problems caused by missing data (Dempster, Laird & Rubin, 1977; Glasser, 1964; Rubin, 1987; Schafer, 1997). The development of these principled procedures is a unique feature of the survey research literature. For the past two decades, the statistical properties of these procedures have been studied and made known. And many have been incorporated into widely accessible software since 1990 (Cool, 2000; Kim & Curry, 1977; Mason, 1999; Raymond & Roberts, 1987; Witta, 2000). Even so, an examination of articles published in 11 education journals for the past 5 years reveals that, in studies where missing data were reported and care was taken to treat missing data, an overwhelming majority of these studies used ad hoc methods, such as listwise and pairwise deletion, to handle missing data. Ad hoc methods are characterized by their failure to take into account the mechanism that led to missing data. Yet the recent APA Task Force on Statistical Inference explicitly warmed against the use of these methods: Special issues arise in modeling when we have missing data. The two popular methods for dealing with missing data that are found in basic statistics packages—listwise and pairwise deletion of missing values—are among the worst methods available for practical applications. (Wilkinson & The APA Task Force on Statistical Inference, 1999, p. 598) Newer and principled methods take into consideration conditions under which missing data occurred. In this paper, we promote the use of three such methods, namely, two singleimputation methods and one based on multiple imputation. These methods are illustrated with real-world data sets, and the results are shown to be different from those obtained from listwise deletion. Key issues in guiding the use of these methods are also discussed. Since these methods have been implemented into statistical software (e.g., SPSS® and SAS®) and should be widely Advances in Missing Data 5 available, their utility by educational researchers is expected to substantially increase. Throughout our illustration and discussion, the focus is on handling cases in which partial information is missing rather than a complete lack of information. The remainder of this paper is divided into nine sections: (1) Problems Caused by Missing Data and Missing Data Treatments, (2) Missing Data Methods Reported in 11 Education Journals, (3) Missing Data Mechanisms, (4) Overview of Five Ad Hoc Missing Data Methods, (5) Principled Single Imputation--FIML, (6) Principled Single-Imputation—SPSS® EM, (7) Multiple Imputation, (8) Importance of Statistical Assumptions, and (9) Implications and Recommendations for Educational Research. 1. Problems Caused by Missing Data and Missing Data Treatments Why are missing data a problem? The most serious concern is that missing data can introduce bias into estimates derived from a statistical model (Becker & Walstad, 1990; Becker & Powers, 2001; Holt, 1997; Rubin, 1987). For example, it is possible that non-respondents might have different response profiles compared to those who responded completely. Thus, the remaining sample is no longer random or representative of the population from which it was randomly drawn. If the researcher chose to draw their conclusions based solely on those who responded, the conclusions would be biased. Furthermore, missing data result in a loss of information and statistical power (Anderson, Basilevsky & Hum, 1983; Kim & Curry, 1977). The elimination of subjects with missing information on one or more variables from the statistical analysis in listwise deletion decreases the error degrees of freedom (df) in statistical tests such as the t. This decrease in turn leads to reduced statistical power and larger standard errors compared to those obtained from complete random samples (Cohen & Cohen, 1983; Cool, 2000). Advances in Missing Data 6 Similar loss of df and statistical power occurs with pairwise deletion, where standard deviations, correlations and covariances are calculated on the basis of ava ilable data on each variable (Glasser, 1964; Raymond & Robert, 1987). As a result, the sample composition differs from variable to variable, and the population to which the results are generalized is no longer clearly defined. Another problem with missing data is that they make common statistical methods inappropriate or difficult to apply (Rubin, 1987). For example, when missing data are present in a factorial analysis of variance the design is unbalanced. Consequently, the standard statistical analysis that is appropriate for balanced designs is no longer appropriate under this condition. Even if data are assumed to be missing in a completely random fashion, the proper analysis is complicated. Multivariate statistical methods, as they are programmed into commercial statistical software, are applicable to complete data sets by default. Finally, valuable resources are wasted as a result of missing data. Time and funding spent on subjects who subsequently leave a study and/or produce missing data represents a loss (Buu, 1999; Holt, 1997). Such loss is a particular concern in longitudinal research, large-scale assessments, high-stake studies, and/or surveys that ask sensitive information or target respondents who are not accustomed to responding to opinions surveys (such as the first generation Hmong immigrants). Efforts to achieve higher response rates and complete profiles from respondents require researchers to allocate additional time and resources to trace cases who failed to respond or those whose responses were incomplete or unusable. These efforts may not always pay off. 2. Missing Data Methods Reported in 11 Education Journals Advances in Missing Data 7 In order to understand how educational researchers currently deal with missing data, we reviewed the quantitative studies published in 11 education journals from 1998 to the summer of 2002. The 11 journals were American Educational Research Journal (AERJ ), Educational Researcher (ER), Journal of Counseling Psychology (JCP), Journal of Educational Psychology (JEP), Journal of Research in Science Teaching (JRST), Journal of Special Education (JSE), Journal of School Psychology (JSP), The Modern Language Journal (MLJ), Research in