The effects of matching criterion contamination on the Mantel-Haenszel procedure

Modem bias detection procedures search for differences in item performance between demographic groups after conditioning on an estimate of the ability intended to be measured by the test. The estimate of ability is typicdly some function of the total test score. Since exarninees with equal test scores are said to be matched on ability, the intemal measure of ability is referred to as the matching criterion. When the test contains one or more biased items, the test score will not be a valid measure of ability. As a result, the matching criterion is said to be contarninated by the biased items. This study consists of a comprehensive examination of the Mantel-Haenszel (MH) procedure in the presence of a contaminated matching criterion. This examination focused on two pnmary issues: (1) assessing the effects of contamination on the MH procedure, and (2) developing alternative DiF detection procedures which are robust to contamination. The results indicate that the presence of contamination has rninor effects on the MH procedure when contamination is small or moderate, but has substantial effects when contamination is large. The effects were related to the proportion of items containing contamination, and increased as the level of DIF in the contaminated items increased. Two solutions to the problern of matching criterion contamination solutions were proposed. First, a procedure was developed that adjusts the obtained MH value to correct for the effects of contamination. The results of a simulation study suggest that the adjustment is effective in general at correcting for the effects of contamination, losing efficiency only under the most severe levels of contamination and the smailer sample size (N = 250). A second solution to the problem of contamination was the proposal of MB-DIF, a new statistic that is theoretically robust to the effects of contamination. The results of a simulation study indicate that the performance of MB-DIF exceeds that of the MH adjustment, particularly when sarnple sizes were large (N = 1000). Under the condition of large sample sizes, MB-DIF was completely robust to the effects of contamination, maintaining power and Type 1 error rates identical to control conditions in which no contamination existed. The performance of MBDIF suffered slightly when sample sizes were small (N = 250), largely due to inflated Type 1 error rates under large levels of contamination. The findings have two implications. First, bias detection analyses should consider the possible magnitude of bias in other items in the test when investigating the magnitude of bias in any given item. Second, adjustment procedures c m control for the majority of the underestimation in DIF statistics when the matching criterion is contaminated. It appears that MB-DiF offers a more effective solution to the problem of matching criterion contamination than adjusting the MH value. First, 1 wouid like acknowledge the time and energy spent by my thesis cornmittee. A special thanks to Dr. Ross Traub for his conceptual and editorial advice, Dr. Phi1 Nagy for the thorough review of the manuscript and those nicely typed comments, Dr. Ruth Childs for valuable advice on making the dissertation better organized, Richard Wolfe for pointing me in the right direction on several instances concerning the mathematical and statistical rigor of the manuscript, and Dr. Hariharan Swaminathan for his insightfùl recommendations. i would also Iike to thank Dr. Shimhiko Nishisato for his early involvement in the proposal of the dissertation. 1 would also like to acknowledge the support 1 received from nurnerous individuals during my doctoral studies. 1 am indebted to Phi1 Nagy for suppiying seemingly endless opportunities for fùnded project research that afforded me the time, resources, and experience necessary to complete my studies in such timely fashion. A huge thanks to Alex Lawson for her partnership over the years, making the late nights in the office bearable, giving me something to look forward to in the evenings (no merlot please! ), and bringing a sense of community to Our group. 1 hope it doesn't end here. Many thanks to Adrienne McRuvie for providing the laughter in my life, and not letîing me forget the big picture. Your lessons will be a part of my life always. Finally, to my parents Doug and Kath for their influence and support throughout this chapter of my life. Individually you both supplied me with the motivation and insight required to get me where 1 am today. Thanks! Table of