Effect size indices for analyses of measurement equivalence: understanding the practical importance of differences between groups.

Because of the practical, theoretical, and legal implications of differential item functioning (DIF) for organizational assessments, studies of measurement equivalence are a necessary first step before scores can be compared across individuals from different groups. However, commonly recommended criteria for evaluating results from these analyses have several important limitations. The present study proposes an effect size index for confirmatory factor analytic (CFA) studies of measurement equivalence to address 1 of these limitations. The application of this index is illustrated with personality data from American English, Greek, and Chinese samples. Results showed a range of nonequivalence across these samples, and these differences were linked to the observed effects of DIF on the outcomes of the assessment (i.e., group-level mean differences and adverse impact). Practitioners and organizational researchers confront a vast number of questions that involve comparing scores on assessment instruments across groups. Are workers more satisfied in organi- zations with empowerment programs? Are successful salespersons more extraverted? Are employees in a multinational organization more satisfied in one country than employees in another? More- over, because of the legal and practical implications of using selection assessments that advantage one group over another, group comparisons may be particularly salient during the hiring process. For all of these comparisons to be meaningful, it is essential that the tests and scales provide equivalent measurement across groups. Equivalent measurement is obtained when individ- uals with the same standing on the trait assessed by the test or scale, but sampled from different groups, have equal expected observed scores (Drasgow, 1984). As such, measurement invari- ance can be examined by a differential item functioning (DIF) analysis using item-response theory (IRT) or with confirmatory factor analytic (CFA) mean and covariance structure (MACS) analysis. The latter method is the focus of this article. Although several articles have proposed various decision rules for determining if measurement nonequivalence exists with MACS analysis (Cheung & Rensvold, 2002; Hu & Bentler, 1999; Meade, Johnson, & Braddy, 2008), these rules generally involve empiri- cally derived cutoffs or statistical significance tests. As such, the analysis does not address the practical importance of observed differences between groups and does not provide users with infor- mation about the effects of nonequivalence on the organizational outcomes of an assessment. In the broader psychological literature, effect size statistics have been proposed to overcome this limita- tion (Cohen, 1990, 1994; Kirk, 2006; Schmidt, 1996). However, effect size indices for CFA evaluations of measurement equiva- lence have not yet been developed. In the present study, we propose such an index and examine its application to real-world data. To illustrate its practical importance, we also demonstrate the effects of measurement nonequivalence on the observed outcomes (e.g., means, adverse impact) of group-level comparisons. This information will enable researchers and practitioners to further evaluate the theoretical and practical importance of observed dif- ferences.