Using a Multidimensionality-Based Framework to Identify and Interpret the Construct-Related Dimensions that Elicit Group Differences

The Roussos-Stout (1996) multidimensionality-based DIF analysis framework emphasizes a substantively-informed selection of items for both the matching and studied subtest based on the dimensions suspected of underlying the test data. By contrast, standard DIF practice encourages an exploratory search for matching subtest items based on purely statistical criteria, such as a failure to display DIF. Using two examples, we demonstrate that these two approaches lead to different interpretations about the occurrence of DIF in a test. It is argued that selecting a matching and studied subtest, as identified using the multidimensional framework, can lead to a more informed understanding of why DIF occurs. Multidimensional Framework 3 Using a Multidimensionality-Based Framework to Identify and Interpret the Construct-Related Dimensions that Elicit Group Differences According to the authors of the Standards for Educational and Psychological Testing (1999), bias occurs when tests yield scores or promote score interpretations that result in different meanings for members of different groups (e.g., race, ethnicity, language, culture, gender, disability, or socio-economic status). Bias is often attributed to construct-irrelevant dimensions that differentially affect the test scores for different groups of examinees. Group differences can also be attributed to item impact. Impact occurs when construct-relevant dimensions differentially affect the tests scores for different groups of examinees. In this case, the item is a relevant measure of the target construct and the difference between the groups reflects a true difference on that construct. Differential item functioning (DIF) studies are designed to identify and interpret these construct-related dimensions using a combination of statistical and substantive analyses. The statistical analysis involves administering the test, matching members of the reference and focal group on a measure of ability derived from that test, and using statistical procedures to identify group differences on test items. An item exhibits DIF when examinees from the reference and focal groups differ, on average, in their probabilities of answering that item correctly, after controlling for ability. The substantive analysis builds on the statistical analysis because DIF items are often scrutinized by expert reviewers (e.g., test developers or content specialists) who attempt to identify the construct-related dimensions that produce group differences. A DIF item is considered biased when reviewers identify some dimension, deemed to be irrelevant to the construct measured by the test, that places one group of examinees at a disadvantage. Conversely, a DIF item displays impact when the dimension that differentiates the groups is judged to be relevant to the construct measured by the test. Considerable progress has been made in the development and refinement of statistical methods for identifying items showing DIF (see reviews by Clauser & Mazor, 1998; Millsap, & Everson, 1993) but the development and refinement of substantive methods designed to aid with the interpretation of these items have lagged far behind (e.g., Bond, 1993; Camilli & Shepard, 1994, Englehard, Hansche, & Rutledge, 1990; Gierl, Bisanz, Bisanz, Boughton, & Khaliq, 2001; Gierl, Rogers, & Klinger, 1999, O’Neill & McPeek, 1993; Plake, 1980; Roussos & Stout, 1996; Standards for Educational and Psychological Testing, 1999; Stout, 2002; Sudweeks & Tolman, Multidimensional Framework 4 1993). The traditional approach—subjecting items flagged with DIF analyses to the scrutiny of reviewers—has not been successful because the interpretations tend to be inconsistent with the DIF statistics or unreliable among reviewers. For example, Camilli and Shepard (1994) reported that, in their experience, as many as half of the items with “large” DIF in any one study might not be interpretable. Angoff (1993) noted: "It has been reported by test developers that they are often confronted by DIF results that they cannot understand; and no amount of deliberation seems to help explain why some perfectly reasonable items have large DIF values" (p. 19). Roussos and Stout (1996) reviewed the DIF literature and claimed, “attempts at understanding the underlying causes of DIF using substantive analyses of statistically identified DIF items have, with few exceptions, met with overwhelming failure” (p. 360). The authors of the Standards for Educational and Psychological Testing (1999) concluded: Although DIF procedures may hold some promise for improving test quality, there has been little progress in identifying the causes or substantive themes that characterize items exhibiting DIF. That is, once items on a test have been statistically identified as functioning differently from one examinee group to another, it has been difficult to specify the reasons for the differential performance or to identify a common deficiency among the identified items. (p. 78) This impasse represents a fundamental problem in the study of group differences using DIF methods. Roussos and Stout (1996) proposed a multidimensionality-based DIF analysis paradigm to bridge the gap between statistical and substantive analyses by linking both to the Shealy-Stout multidimensional model for DIF (Shealy & Stout, 1993). The first stage is a substantive analysis where DIF hypotheses are generated. The second stage is a statistical analysis where the DIF hypotheses are tested. By combining statistical and substantive analyses in a multidimensional framework, researchers and practitioners can begin to systematically identify and interpret the construct-related dimensions that produce group differences using DIF methods. The purpose of this paper is twofold: In the first section, we describe the Roussos and Stout (1996) DIF analysis framework. In the second section we illustrate, using two examples, how the DIF analysis framework can lead to more interpretable results about the dimensions that produce Multidimensional Framework 5 group differences when compared with the traditional approach to DIF detection. We conclude with a summary and we highlight some implications for practice. DIF Analysis Framework: An Overview Roussos and Stout (1996) proposed a multidimensionality-based DIF analysis framework to link substantive and statistical analyses so researchers and practitioners can begin to systematically identify and study the sources of DIF. The DIF analysis framework is rooted in the Shealy and Stout (1993) multidimensional model for DIF (MMD), which serves as a theoretical basis for understanding how DIF occurs. A dimension is a substantive characteristic of an item that can affect the probability of a correct response. The main construct the test is intended to measure is the primary or target dimension. The MMD is based on two assumptions: (a) DIF items elicit at least one secondary dimension in addition to the primary dimension and (b) a difference exists between the two groups of interest in their conditional distributions on the secondary dimension, given a fixed value on the primary dimension. Roussos and Stout (1996) interpreted the secondary dimensions further. The secondary dimensions are auxiliary if they are intentionally assessed as part of the construct on the test, which implies the construct of interest contains multiple dimensions. DIF caused by auxiliary dimensions is benign (reflecting impact). Alternatively, the secondary dimensions are nuisance if they are unintentionally assessed as part of the construct on the test. DIF caused by nuisance dimensions is adverse (reflecting bias). On a test of mathematics achievement, for example, knowledge of mathematics might be a primary dimension, critical thinking might be an auxiliary secondary dimension, and testwiseness (i.e., using strategies to select the correct answer based on knowledge of test item characteristics) might be a nuisance secondary dimension. If a DIF item favors females and this difference can be attributed to the critical thinking auxiliary secondary dimension, when considered in isolation from the mathematics primary dimension, then DIF is considered benign. Alternatively, if a DIF item favors males and this difference can be attributed to the testwiseness nuisance dimension, when considered in isolation from the mathematics primary dimension, then DIF is considered adverse. The Roussos-Stout DIF analysis (1996) framework is a two-stage procedure built on the foundation provided by the MMD. The first stage is a substantive analysis where the dimensional Multidimensional Framework 6 structure of the test is evaluated and, based on this structure, where DIF hypotheses are generated. To decide whether the data contain distinct dimensions, organizing principles are used to identify single items or bundles of items that share certain characteristics. Four different organizing principles have been used to identify dimensions on tests (Ackerman et al., 2003; Douglas et al., 1996; Gierl et al., 2001; Roussos & Stout, 1996). First, test specifications can guide the assessment of dimensionality. Test specifications outline the achievement domain and help test developers obtain a representative sample of items from this domain. The specifications also guide item writing and help structure the final form of the test based on the content and cognitive domain that the test is designed to measure. Thus, a thorough analysis of the content areas measured by the test and the cognitive skills required by the examinees to solve the items may help identify a subsets of items that measure distinct dimensions associated with these content areas and cognitive skills (e.g., Gierl et al., 2001; Oshima, Raju, Flowers, & Slinde, 1996). Second, a content analysis can guide the assessment of dimensionality. For example, content specialists can review items and identify dimensions based on specific item content. A content analysis is guided by the