Comparability of paper and computerized non-cognitive measures: A review and integration

This study reviewed the literature on equivalence of paperand computer-administered noncognitive assessments. We summarize the designs available for analysis of such studies and previous research. Although dozens of studies have been completed, only 6 studies with 41 effects and a total sample size of N=760 met our criteria. Our meta-analysis of these results did not support hypothesized computerization effects. DRAFT Comparability Review 3 Comparability of paper and computerized non-cognitive measures: A review and integration Increasingly, many HR services involving surveys, job-attitude measures and employment testing are being implemented using computers and computer networks such as the Internet. These services are being migrated to automated delivery channels for the same efficiency reasons that drive banks to offer on-line banking and organizations to offer on-line HR benefits information: properly implemented automated services are more available, more convenient, more efficient, and less costly to operate. As more web services are offered, organizations gain even greater economies of scale and cost savings become more assured. Thus this automation process provides greater value. And as a result, more surveys, questionnaires, and tests are being administered by computer and this trend is likely to continue. But what if the automated measures are not comparable with their paper-and-pencil counterparts? This concern surfaced with the very first computerized assessments (e.g., see Vinsonhaler, Molineaux & Rodgers, 1968) and is ensconced in the American Psychological Association’s Guidelines for Computer-Based Tests and Interpretations (APA, 1986). The Guidelines state that computerized and paper-and-pencil measures are to be considered comparable only after appropriate research has established this empirically. Specifically: When interpreting scores from computerized versions of conventional tests, the equivalence of scores from computerized versions should be established and documented before using norms or cutting scores obtained from conventional tests. Scores from conventional and computer administrations may be considered equivalent when (a) the rank orders of scores of individuals tested in alternative modes closely approximate each others, and (b) the means, dispersions, and shapes of the score distributions are approximately the same, or have been made DRAFT Comparability Review 4 approximately the same by rescaling the scores from the computer mode. (APA, 1986) This paper has two goals. First, it will review the research designs used to assess comparability and describe the strengths and weaknesses for answering the research question, “Are the paperand computer-based versions comparable? ” Researchers should choose the best design possible. Different designs also have different implications for meta-analysis. A second goal is to provide an empirical meta-analysis of the comparability of measures other than cognitive ability measures. Cognitive ability measures have previously been reviewed (see Mead & Drasgow, 1993) and relatively frequently studied. Yet non-cognitive measures are probably more often used and encompass a greater diversity (of traits, factor structures, scoring methods, item types, etc.). Thus it is important to review the literature on these non-cognitive measures to determine the accumulated knowledge. Research on Comparability Numerous reasons have been proposed to doubt comparability. In the case of computerized surveys, attitude measures, and questionnaires, respondents may have different privacy perceptions regarding computerized and paper forms. Individuals can generally see whether their identity is indicated on paper forms while it is generally unclear whether on-line responses could be traced back to identify themselves; often on-line surveys describe privacy assurances but these may not be trusted by the respondent. Perceptions of anonymity may, for example, be reinforced if the survey does not require an individual login, is available from anywhere, and is hosted by a third party. On the other hand, if the respondent must supply their employee number to access the survey, which is hosted on the company intranet, then even assurances of privacy may not dispel suspicions of skeptical respondents. DRAFT Comparability Review 5 Differences between paperand computer-based tests in responding, such as ability to omit or change answers, has been suggested as a factor affecting comparability (Spray, Ackerman, Reckase, & Carlton, 1989; King & Miles, 1995). Oswald (19xx) has shown that undergraduate volunteers respond differently . However, Mead and Counssins-Read (19xx) used a different design and found very different results. Also, respondents using computers may have a different experience due to lack contact with an administrator. For example, it is common for respondents to personality questionnaires to have occasional questions. In paper testing, these individuals may receive clarification of the instructions from an administrator. These individuals would have to proceed on an automated test with their idiosyncratic understanding of the instructions. Further, given that web-based measures may often be accessible from anywhere, respondents to computerized measures may complete the measures in an inappropriate (or, at least, non-standardized) time or place. Mead and Coussons-Read (2002) reported that “a surprising number” of respondents to a web-based personality measure completed the instrument during the late night and early morning hours. The aggregate effects of such issues seem sufficient to exercise caution regarding the comparability of paper and computerized forms. And comparability is important for I/O psychologists in at least two situations. First, many testing programs require paper forms for some uses. For example, paper booklets may be used during occasional computer failures, when group testing precludes the use of computers, or when testing in environments without computers (e.g., job fairs, factory floors, etc.). It is also very common for the validity evidence and other documentation of an instruments psychometric properties to be based on a paper version of the instrument. DRAFT Comparability Review 6 Previous reviews [This section is currently under construction] Comparability Research Designs Several research designs have been chosen by researchers in this area. In this section, we describe some of the strengths and limitations of designs. We will also examine the degree to which different designs can be meta-analyzed and the data required in original reports to allow inclusion in a meta-analysis. Table 1 presents a simple taxonomy of study designs. The broadest difference between designs is whether they are between-subjects or within-subjects. Between-Subjects Designs. Between-subjects designs are those in which different groups of participants respond to the paper and computerized measures. Typically, the means of the paper and computerized measures are compared (using t-tests, ANOVA/ANCOVA, etc.) and significant differences are taken as an indication of non-equivalence. Such studies seem to follow directly from typical experimental designs such as drug studies and are readily “metaanalyzable. ” An obvious methodological issue is the equivalence of the groups; if the groups are not formed on the basis of random assignment then any results could be caused by the medium of administration, the group differences, or some interaction of these two factors. This is particularly a concern if the different samples of convenience define the computerand paperadministration groups. For example, if the tests were administered on computer to workers at a new facility and on paper to workers at an older facility, any effect found could well be due to differences in the characteristics, context, administration procedures, etc. at these two facilities rather than the computerization. DRAFT Comparability Review 7 Even if the two groups are randomly equivalent, between-subjects designs do not directly test the hypothesis of measurement equivalence because it is possible for the mean test scores of the computerized and paper versions to be exactly equal while the rank-order of the individuals in the groups entirely different. In such a situation, the versions are definitely not equivalent but because the between-subjects design does not evaluate the equivalence of the rank-orderings of the versions, no difference would be found. Similarly, it would be possible for the ran-order of individuals in the two groups to be identical while the mean scores could be very different. Because between-subjects designs cannot assess similarity of rank-orderings in the two groups, they are a very incomplete test of equivalence. Another significant methodological issue is more subtle; mean comparisons such as ANOVA are not designed to test equivalence. Standard statistical techniques developed to compare experimental and control groups always test non-equivalence. And in the logic of standard hypothesis testing, a failure to reject the null hypothesis of no differences is not at all the same as demonstrating the null hypothesis. Thus equivalence of the paper and computerized groups is typically “shown” by failing to reject the null hypothesis of no differences but this is a misuse of the statistical methods (see Cohen, 1994). A primary result of such misuse is that the probabilities of Type I and II errors changes radically when the statistical tests are used this way. Studies that misuse significant testing in this way must have a high degree of power; otherwise, the study is likely to find equivalence regardless of the true equivalence or non-equivalence. The primary influence on power is these designs is sample size and thus such studies must have large sample sizes in order to produce results that are at all meaningful. As an alternative to traditional statistical hypothesis testing