Stealth Marking for Improved Quality Assurance in The Ethics and Law Assessment
Creators: name: Thewlis, David
name: Coleman, Jamie
Publisher: name: Zenodo
Year: 2019
Subjects: Medical Education
Assessment
Rubric
Reliability

description: <strong><em>Background and purpose</em></strong> Medical professionalism and ethics are difficult areas difficult to assess prior to graduation. The University of Birmingham MBChB program includes an Ethics and Law module assessed by a written assignment in the students’ final year. Historically this assignment was assessed by a single examiner with sample moderation for quality assurance. Given this is the last opportunity for students to demonstrate professionalism in a written assessment it was felt that enhanced quality assurance measures should be put in place. The risk of construct-irrelevant variance is ever present in written assessments (1). The use of marking rubrics and blind marking have both been put forward as methods to reduce bias in marking (2,3). The use of a second marker is recognised as a way to increase the reliability of a written assessment(4) however this also decreases the feasibility of the assessment. The investigating team decided to pilot an electronic mark sheet which would provide some of benefits of a second marker without the need for increased work, whilst incorporating the use of blind marking and the a marking rubric. <strong><em>Methodology</em></strong> A marking rubric was devised for the Ethics and Law Essay. Within the Excel mark sheet examiners were asked to pick specific anchor statements from the rubric for entry into drop down boxes. These were concatenated to create generic feedback for students in addition to the bespoke feedback examiners would provide. Each anchor statement was assigned a value which was hidden from examiners. These values corresponded to either an “Excellent”, “Good”, “Pass”, Borderline Fail”, or “Poor” grade. The marking rubric provided to examiners displayed the approximate percentage each of these grades would receive overall, but did not inform them of the weighting of each domain. Domains included “background reading and research”, “relevance of case” amongst others. Examiners provide an overall percentage mark to each essay which will be termed the “awarded mark”. The “stealth mark” based upon the chosen anchor statements was visible to the investigators only. These data were compared to examine the level of inter-rater agreement between markers and stealth marks. Inter-rater agreement was defined as the examiner awarding a mark within 5% of that produced by the stealth marking formula. Marks for essays were excluded if the examiner had manipulated anchor statements so that they no longer clearly mirrored those in the original rubric. Where anchor statements had been manipulated, but with only subtle changes the investigators entered the value matching the closest anchor statement. The author’s essay marks were excluded due to a lack of blinding to the process. 319 essay marks were analysed for agreement. 190 of these were marked by relatively inexperienced markers (those with 1 or few years of marking). <strong><em>Results</em></strong> Intra-rater agreement (agreement between awarded mark and stealth mark) varied from 0.5- 1.0 between different examiners. There seemed to be no difference between markers based upon level of experience. Mean agreement within the cohort of inexperienced markers was 0.78 whilst within the cohort of experienced examiners agreement was 0.80. The agreement value (actual mark given – stealth mark) showed a range from -16.5 to +16 with a standard deviation of 4.4. 80% of marks demonstrated agreement when analysed across all markers, but with evidence of significant outliers – at either end of the range.
description: {"references": ["Allen D, Tanner K. Rubrics: tools for making learning goals and evaluation criteria explicit for both teachers and learners. CBE life sciences education 2004;5(3):197-203.", "Brooks V. Double Marking Revisited. British Journal of Educational Studies 2004;52(1):29-46.", "Rezaei AR, Lovorn M. Reliability and validity of rubrics for assessment through writing. Assessing Writing 2010;15(1):18-39.", "The Research Methods Knowledge Base. Types of Reliability. 2006; Available at:\u00a0http://www.socialresearchmethods.net/kb/reltypes.php. Accessed 07/03/19, 2019."]}