Do Employment Test Publisher Manuals Provide Accurate Information On Test Validity ?

This paper has two goals. First, we discuss the issue of publication bias and explain why it presents a problem for organizational and human resources research. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (Trim and Fill) that has been developed in the medical literature but which is largely unfamiliar to human resources, organizational behavior, and management researchers . We demonstrate the Trim and Fill approach to conducting publication bias analyses by applying it to validity information reported in three test publishers’ technical manuals. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. Our application of the Trim and Fill publication bias method to twelve validity distributions (five from one publisher, three from a second publisher, and four from a third publisher) found evidence of either no bias or minimal bias for most of the distributions. However, in the case of one publisher, the level of bias in two of three cases was serious enough to substantially change validity estimates. Publication bias in test vendor manuals 2 Publication bias is the term used to refer to the fact that not all completed studies on a topic make their way into the published literature, and that these studies are likely to be systematically different from those that do appear in the literature. When the results of publicly available research differ from the results of all the research that has been done in an area, readers and reviewers are in danger of being misled. Publication bias can have powerful consequences, particularly when a dangerous or ineffective practice is viewed as effective or safe because of selective publication of results. For example, Merck pharmaceutical recently recalled Vioxx, its popular arthritis drug. While Merck maintained that it recalled Vioxx as soon as the data indicated a high prevalence of cardiovascular problems among those who took this drug for more than 18 months, media reports claimed that Merck hid adverse event evidence for years (Wall Street Journal, November 1, 2004). Similarly, the attorney general of New York State, filed a 2004 lawsuit against GlaxoSmithKline, in which it was claimed that they concealed data about the lack of efficacy, and about the increased likelihood of suicide associated with the use of Paxil by children and teenagers (NY vs. GlaxoSmithKline). In the human resources and organizational behavior literature, the consequences of publication bias are not likely to be fatal; nevertheless we can think of several areas where they can be quite serious. One of the most important is the area of employment test validities. Specifically, there are important negative consequences for employers, and job applicants, if an invalid test is falsely viewed as valid, or if a test with low validity is viewed as a having high validity. Unfortunately, we know next to nothing about the likelihood that publication bias is affecting the published information on employment test validities; in fact we know very little about the extent to which publication bias threatens the validity of most research literatures in human resources or organizational behavior. While there is a burgeoning literature examining publication bias in healthcare, (Dickersin, 2005; Halpern & Berlin, 2005) and while the topic has received increased attention among some areas of psychological research (Bogg & Roberts, 2004; Cooper, 2003; Mezulis, Abramson, Hyde & Hankin, 2004; Moyer, Rounds & Hannum, 2004; Robbins, Lauver, Le, Davis, Langley & Carlstrom, 2004; Smith, McCullough & Poll,2003; Vitaliano, Zhang & Scanlan, 2003), the potential problems posed by publication bias have generally been ignored in the personnel, human resources, organizational behavior and management literatures. To the extent that publication bias has been addressed in our literature, the outdated failsafe N method, also known as the “file drawer problem” method is used. This method has been shown to be a poor procedure for assessing publication bias (Becker, 2005) and more accurate and powerful approaches have been offered (Duval, 2005; Hedges & Vevea, 2005; Sterne & Egger, 2005; Sutton & Pigott, 2005). The current paper has a two-fold purpose. First, we describe two methods for the assessment of publication bias. After reviewing and critiquing the familiar failsafe N, or file drawer analysis method, we describe a newly introduced procedure to assess publication bias. This procedure, called Trim and Fill (Duval Publication bias in test vendor manuals 3 & Tweedie, 2000a, 2000b), has been used widely in the healthcare literature, but is largely unfamiliar to, and therefore unused by, human resources, organizational behavior, and management researchers. Second, we illustrate the operation of the Trim and Fill procedure by using it to explore the possibility that criterion-related validity information in test publisher manuals is subject to publication bias. Specifically, we apply Trim and Fill to sets of employment test validities drawn from the manuals of three test publishers. Publication Bias and Procedures for Its Detection Publication bias is classically defined as the tendency to publish studies depending on the magnitude, direction or statistical significance of their results. There are several possible causes for the suppression of negative, low, or statistically non-significant research results; most of them are clearly relevant to the domain of test validities. We outline these here. There is considerable evidence that one cause of bias is that researchers are not likely to submit negative results for publication (Dickersin, 2005). We suggest that researchers also are not likely to share weak (poor) validity results with the test publisher. For example, a researcher who has found that a test in use by his or her company has poor validity may withhold these data due to concerns about employment litigation or embarrassment to the organization. There also is some evidence that editorial policy, at least in some journals, favors the publication of significant results (Dickersin, 2005; Greenwald, 1975). Similarly, test manual publishers might favor the reporting of positive, statistically significant validities. This could be because they view these as more important or more interesting, or because they are concerned that insignificant, low or negative validities present the test in a poor light. Rosenthal Failsafe N (the file drawer problem) Rosenthal (1979) introduced what he called the “file drawer problem. ” His concern was that some non-significant studies may be missing from an analysis (i.e., hidden in a file drawer) and that these studies, if included, would nullify the observed effect. By “nullify,” he meant to reduce the cumulated effect to a level where it was not statistically significantly different from zero. Rosenthal suggested that rather than speculate on whether the file drawer problem existed, the actual number of studies that would be required to nullify the effect could be calculated. Cooper (1979) called this number the failsafe sample size or failsafe N. If this number is relatively small, then there is indeed cause for concern. However, if this number is large, we can be confident that the mean observed correlation, while possibly inflated by the exclusion of some studies, is nevertheless not zero. The failsafe N method has been used to assess the likelihood of publication bias in several human resources and organizational behavior meta-analyses including Jenkins, Mitra, Gupta and Shaw (1998), Mitra, Jenkins and Gupta (1992) and Rhoades and Eisenberger (2002). Publication bias in test vendor manuals 4 This approach is limited in two important ways (Becker, 2005). First, it assumes that the correlation in the hidden studies is zero, rather than considering the possibility that some of the studies could have shown an effect in the reverse direction or an effect that is small but not zero. Therefore, the number of studies required to nullify the effect may be different than the failsafe N, either larger or smaller. Second, and more fundamentally, this approach focuses on statistical significance rather than practical or substantive significance. That is, it may allow one to assert that the mean correlation is not zero, but does not address what the unbiased correlation is and whether it remains of useful size after the missing studies have been included. Consider an employer choosing between two tests A and B offered by different test publishers. The validity information for test A suggests that the test has a mean validity of . 25 while the validity information for test B shows a mean validity of . 20. If there is no publication bias, the employer would choose test A, all other things (cost, ease of administration, etc.) being equal. However, if publication bias is suspected, one would like to know what the validity of tests A and B is likely to be in the absence of the bias. Knowing that it takes 80 file drawer studies to nullify the validity of test A and 100 file drawer studies to nullify the validity of test B in no way helps to determine what the validity of the tests are in the absence of publication bias. In acknowledgment of the fact that statistical significance levels are not typically as of much concern as are effect sizes, Orwin (1983) extended the idea of the failsafe N to effects sizes, and reformulated the question as “how many effect sizes averaging a particular value would be needed to reduce an observed mean effect size to a level at which it was no longer theoretically or practically significant. Orwin’s variant has also been used in I/O and OB meta-analyses, (a good example is McNatt, 2000). Although Orwin’s method is an improvement on the original Rosenthal method, in that it incorporates information about effect si