Validation of a Rubric to Assess Physiology Exam Multiple Choice Questions to Create Fair and Equitable Assessments
Authors: Bruce Erin , Sullivan Rebecca , Lopez Marisol 
Published: May 2024

Inequities and achievement gaps exist for students from groups underrepresented in medicine (URiM) and for English language learners (ELLs). This is particularly evident with multiple choice assessments which are prevalent in health professions and medical education. Increasing the linguistic accessibility of test items by simplifying the language used while keeping the content diffculty was perceived as helpful to ELLs in an undergraduate Chemistry course (Lee & Orgill, 2022). Following best practices when writing exam questions provides consistent and valid measures of student comprehension. Across institutions we face similar challenges when forming and assessing multiple choice questions for our physiology exams. Some major flaws include long complex options; using vague terms; negatively structured stems; grammatical cues; word repeats. Lopez (2023) found that 56% items contained at least one flaw. The most common flaws identified were those in which the items were not approximately the same length and item options that did not have parallel grammatical form and structure. In order to address these issues in an objective way, we collectively modified the previously detailed rubric (Lopez, 2023) for identifying common flaws based on the “Item-Writing Guide” from the National Board of Medical Education (NBME®) and one formed in the context of general chemistry by Breakall et al. (2019). Twenty-six questions from physiology exams at three different institutions were evaluated by three faculty using the newly created rubric. After analyzing the questions, we returned as a group and compared our individual ratings, clarified any discrepancies and modified the rubric accordingly. We will conduct statistical analysis to assess our inter-rater reliability score before using this rubric to analyze the full question banks for physiology exams at each institution. This process is not limited to physiology educators since it can be utilized by teachers in any program or discipline. The significance of the work will not only benefit our validation of learning but it will also address specific equity issues by eliminating flaws that provide irrelevant diffculty or benefit test wise students. Furthermore, this rubric is also helpful in generating new questions for future exams, ensuring equitable assessments. References: Breakall, J., Randles, C., & Tasker, R. (2019). Development and use of a multiple-choice item writing flaws evaluation instrument in the context of general chemistry. Chem. Educ. Res. Pract. 20(369), 369-382. DOI: 10.1039/c8rp00262b. Lee, E.N. & Orgill, M. (2022). Toward equitable assessment of English language learners in general chemistry: Identifying supportive features in assessment items. J Chem. Educ. 99, 35-48. DOI: 10.1021/acs.jchemed.1c00370. Lopez, M.E. (2023). Assessing multiple-choice questions based on language precision and best practices to promote equity in the Dental Physiology course. Physiology 38(S1). DOI: 10.1152/physiol.2023.38.S1.5731455. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.