DIF in a Large Scale Mathematics Assessment: The Interaction of Gender and Ethnicity.

A study was conducted to investigate how differential item functioning (DIF) is associated with student characteristics such as jender and ethnicity for the total group and to understand the pattern and nature of existing group differences by conducting DIF analyses separately within gender and ethnicity. The research is unique in that it undertakes a two-way DIF approach by taking into account the interaction of gender and ethnicity. Data were from the Delaware Student Testing Program. The multiple choice items for grades 3, 5, 8, and 10 were used, for a total group of 34,544 students. Results of the Mantel Haenzel DIF analyses found an approximately equal number of DIF items favoring the reference and focal groups across all four grade levels. There was some statistical evidence of DIF in all grades, but the DIF was more balanced out by gender and ethnicity in the higher grades (8 and 10). A number of gender and ethnic DIF items that were previously undetected in a total analyses were flagged when two-way procedures were applied. It is clear that two-way DIF analyses offer a more complete and comprehensive approach to DIF detection, superior to the traditional cme-way DIF analysis approach and particularly useful in large-scale testing programs. (Contains 30 tables and 32 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document. 1 PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) U.S. DEPARTMENT OF EDUCATION Office Educational Research and Improvement ED TIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received from the person or organization . originating it. 0 Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. DIF in a Large Scale Mathematics Assessment: The Interaction of Gender and Ethnicity Yanling Zhang Program of Educational Research and Evaluation Ohio University Paper presented for the annual American Educational Research Association conference, April 2002, New Orleans. 2 BEST COPY AVAILABLE Two-Way DIF 2 Objectives The majority of recent research on mathematics performance differences, bias and/or DIF, has been focused on gender (e.g., Cole, 1997; Doolittle & Cleary, 1987; Jacklin, 1989; Linn & Hyde, 1989; Willingham & Cole, 1997), while a far smaller amount of research has been focused on ethnicity (O'Neil & McPeek, 1993; Schmitt & Dorans, 1990; etc.). Almost without exception, the studies concerning mathematics achievement found thus far on DIF have investigated gender separately from ethnicity, or vice versa. Virtually all studies conducted on DIF procedures have typically been based on aggregated gender and ethnicity data. This marginal DIF analysis ignores potential interactions between gender and ethnicity, interactions that may be important. Analyzing DIF at the global level does not serve the purpose of illuminating actual gender and ethnic performance differences. The purpose of this observational study is twofold: (1) to investigate how DIF is associated with student characteristics such as gender and ethnicity for the total group, and (2) to better understand the pattern and nature of existing group differences by conducting DIF analyses separately within gender and ethnicity. In other words, this research is unique in that it undertakes a 2-way DIF approach by taking into account the interaction of gender and ethnicity. Background and Theoretical Framework To date, many states have mandated standardized achievement tests for elementary and secondary school students. Such statewide achievement tests often have high-stakes attached to their use. Ensuring a test is free from biased items is critical to the validity of the test. Test bias occurs when performance on a test requires sources of knowledge different from those intended to be measured, causing the test scores to be less valid for a particular gyoup (Camilli & Shepard, Two-Way DIF 3 1994). One way to screen for bias items is to perform statistical procedures to detect differential item functioning (DIF). According to Hambleton, Swaminathan, and Rogers (1991), "an item shows DIF if individuals having the same ability, but from different groups, do not have the same probability of getting an item right. " (p. 110). Over the past thirty years, there has been a considerable shift in what bias means. Because of the negative connotations of the term bias, in the mid-80s, a more neutral term called differential item functioning (DIF) was proposed (Holland & Thayer, 1988), referring to items that affect performances of comparable groups differently on the trait in question. For any given large-scale assessment, DIF evaluation for a given test is suggested as a standard procedure as stated in the Standards for Educational and Psychological Testing (AERA, APA, NCME, 1985): "operational use of a test will often afford opportunities to check for group differences in test performance and to investigate whether or not these differences indicate bias" (3:10). Among all subject areas to be assessed, mathematics is one of the most studied areas because it is an important component of fundamental education for any society. Mathematics achievement assessment also has serious consequences over students' subsequent schooling and career choices. A substantial amount of research has been conducted with respect to gender difference and/or DIF in mathematics (Carlton & Harris, 1992; Cole, 1997; Fennema & Carpenter, 1981; Jacklin, 1989; Linn & Hyde, 1989; Linn & Petersen, 1985; Maccoby & Jacklin, 1974; Sheuneman & Grima, 1997; Willingham & Cole, 1997, etc.). Previous research demonstrates that standardized achievement tests and college admissions tests given over a wide range of ages and educational levels have recognized a rough pattern of male-female difference. Overall, at the outset of schooling, very little difference has been observed between elementary girls and boys. Gender performance discrepancy begins to emerge in late elementary school. Two-Way DIF 4 The largest differences exist in the subject area of mathematics, where males appear more proficient than females, particularly in secondary school and beyond. In recent years, evidence has been collected, including meta-analyses, supporting the claim that such group mean difference is narrowing. In an attempt to identify possible patterns, some researchers have tried to relate the score differences to the content of the tests (Aiken, 1986-87; Doolittle, 1989; Harris & Carlton, 1993; O'Neil & McPeek, 1993; Ryan & Fan, 1996). Findings indicate that boys are prone to experience less difficulty in geometry, and problem-solving items such as word problems and applied mathematics items whereas girls tend to perform better with conceptual, algebra questions, and items with symbols. In addition to group mean scores, emphasis has been placed on the distribution and variability of scores (Benbow, 1990; Fan, Chen, and Matsumoto, 1997; Feingold, 1992; Willingham and Cole, 1997). Research findings in this field have found that males tend to have a more spread out score distribution, with more numbers of examinees dispersed towards the higher and lower ends of the score range; females are more clustered to the center (or the mean) of the score range. Methods Instrumentation The Delaware Student Testing Program (DSTP) is a mandated statewide assessment program. Data from the 1999 DSTP mathematics section were used for global and 2-way DIF analyses. The test consists of two sections: The Stanford Achievement Series, 9th edition (SAT9) of the abbreviated version in mathematical problem solving items (30 items) and Delaware developed items (MDE) (37 items). There are three item formats, MC, short answers (SA), and extended constructed response (CR) items. Two-Way DIF 5 The present study was set within the context of gender (male and female), ethnicity (Black, Asian, Hispanic, and White), and grade level (grades 3, 5, 8, and 10). Only the multiple choice format items were considered for analyses. The demographic frequency distributions are listed in Tables 1 and 2 below.