Running Head: IMPROVED SCORING OF THE IAT SCORING PROCEDURES TO IMPROVE IMPLICIT ASSOCIATION TEST MEASURES

In reporting Implicit Association Test (IAT) results, researchers have most often used scoring conventions described in the first publication of the IAT (A. G. Greenwald, D. E. McGhee, & J. L. K. Schwartz, 1998). Demonstration IATs available on the Internet have produced large data sets that were used here to evaluate alternative scoring procedures. Candidate new algorithms were examined in terms of their (a) correlations with parallel self-report measures, (b) internal consistency, (c) sensitivity to known influences on IAT measures, and (d) resistance to known extraneous influences. The best-performing measure incorporates data from the IAT’s practice trials, uses a metric that is calibrated by each respondent’s latency variability, and includes a latency penalty for errors. This new algorithm strongly outperforms the earlier (conventional) procedure. Acknowledgment Note. The revised scoring procedures described in this report are hereby made freely available for use in research investigations. An electronic version of this manuscript, along with SPSS syntax for computing IAT measures using the improved algorithm can be obtained at http://faculty.washington. edu/agg/iat_materials. htm/. However, the improved scoring procedures described in this report (patent pending) should not be used for commercial applications nor should they or the contents of this report be distributed for commercial purposes without written permission of the authors. This research was supported by three grants from National Institute of Mental Health, MH-41328, MH-01533, and MH-57672. The authors are very grateful to Mary Lee Hummert, Kristin Lane, Deborah Mellott, Laurie Rudman, and Eliot Smith for helpful comments on an earlier draft. Correspondence concerning this article should be addressed to Anthony G. Greenwald, Department of Psychology, University of Washington, Box 351525, Seattle, WA, USA, 98195-1525. Electronic mail may be sent to agg@u.washington. edu. Greenwald, Nosek, & Banaji: Improved Scoring of the IAT Draft of October 22, 2002 Page -2Scoring Procedures to Improve Implicit Association Test Measures The Implicit Association Test (IAT) provides a measure of strengths of automatic associations. This measure is computed from performance speeds at two classification tasks in which association strengths influence performance. The apparent usefulness of the IAT may be due to its combination of apparent resistance to self-presentation artifact (Banse, Seise, & Zerbes, 2001; Egloff & Schmukle, in press; Kim & Greenwald, 1998), its lack of dependence on introspective access to the association strengths being measured (Greenwald, Banaji, Rudman, Farnham, Nosek, & Mellott, 2002), and its ease of adaptation to assess a broad variety of socially significant associations (see overview in Greenwald & Nosek, 2001). The IAT’s measure, often referred to as the IAT effect, is based on latencies for two tasks that differ in instructions for using two response keys to classify four categories of stimuli. Table 1 describes the seven steps of a typical IAT procedure. The first IAT publication (Greenwald, McGhee, & Schwartz, 1998) introduced a scoring procedure that has been used in the majority of subsequently published studies. The features of this conventional algorithm (see Table 7) include (a) dropping the first two trials of trial blocks for the IAT’s two classification tasks (Blocks 4 and 7 in Table 1), (b) recoding latencies outside of lower (300 ms) and upper (3,000 ms) boundaries to those boundary values, (c) logtransforming latencies before averaging them, (d) including error-trial latencies in the analyzed data, and (e) not using data from respondents for whom average latencies or error rates appear to be unusually high for the sample being investigated. The main justification for originally using these conventional procedures was that, compared with several alternative procedures often used with latency data, they typically yielded larger statistical effect sizes. Previous theoretical and methodological analyses of latency measures have addressed methods of dealing with problems that occur in the form of speed-accuracy tradeoffs (e.g., Wickelgren, 1977; Yellott, 1971), age-related slowing (e.g., Faust, Balota, Spieler, & Ferraro, 1999; Ratcliff, Spieler, & McKoon, 2000), and spurious responses that appear as extreme values (or outliers – Miller, 1994; Ratcliff, 1992). Remarkably, research practice in cognitive and social psychology has been no more than mildly influenced by this previous methodological work. This limited influence is perhaps explained by three practical considerations: First, some of the methodological recommendations are costly to use – for example, several hours of data collection with each subject may be needed to obtain data sets from which individual-subject speed-accuracy tradeoff functions can be constructed. Second, journal editors and reviewers rarely insist on the more painstaking methods. Third, researchers who use the more sophisticated (and painstaking) methods are rarely rewarded for the extra work – conclusions based on those methods often diverge little from those based on simpler methods. The conventional scoring procedure for the IAT has not been subject to systematic investigations of psychometric properties. Additionally, the conventional procedure lacks any compelling theoretical rationale (Greenwald, 2001). Consequently, the authors welcomed a fortuitous opportunity to compare the conventional procedure with alternatives. This opportunity arose through the operation of an educational website (http://www.yale.edu/implicit/) at which several IAT procedures had been made available for demonstration use by drop-in visitors. This article first describes the IAT website and then presents a series of studies that were designed to evaluate candidate alternative scoring procedures for IATs that operated on the website. The investigated scoring methods included (a) transformations of latency measures, (b) procedures for dealing with extreme (slow and fast) responses, (c) replacement (penalty) schemes for error trials, and (d) criteria for identifying a respondent’s data as unfit for computing IAT measures. The article concludes by recommending a replacement for the conventional IAT scoring algorithm. Greenwald, Nosek, & Banaji: Improved Scoring of the IAT Draft of October 22, 2002 Page -3The rationale for interpreting the IAT’s association strength measures as indicators of social-cognitive constructs such as implicit attitude or implicit stereotype rests on theoretical definition of those constructs in terms of concept-attribute associations. This theoretical conception has been described by Greenwald, Banaji, Rudman, Farnham, Nosek, & Mellott (2002). The Yale IAT Website The Yale IAT website was intended to function as the Internet equivalent of an interactive exhibit at a science museum. The site was designed to allow web visitors to experience what the authors and many laboratory subjects have experienced – inability to control the manifestations of automatic associations that are elicited by the IAT method. Drop-in visitors could take demonstration versions of IATs that had been in laboratory use for 2–4 years. Within five to ten minutes, a visitor to the website could complete a measure of implicit attitude or stereotype, after optionally responding to some items that requested demographic information and explicit (self-report) measures of the target attitude or stereotype. Unlike laboratory IATs, the website IATs provided respondents with a summary interpretation of their test performance by characterizing it as showing ‘strong’, ‘medium’, ‘slight’, or ‘little or no’ association of the type measured by each test. Respondents could also inspect distributions of summary results for large numbers of previous respondents. Amplifying the usual debriefing procedure of an experiment, the website also provided answers to numerous questions concerning the IAT’s methods and interpretations, including a discussion of the distinction between the implicit prejudice that the IAT sometimes measures and the more ordinary meaning of (explicit) prejudice. Approximately 1.2 million tests were completed at the Yale IAT website between October 1998 and May 2002, when the present analyses were begun.