Title: Video-based Examiner Score Comparison and Adjustment (VESCA): a complex intervention to improve the dependability of Objective-structured clinical examinations (OSCEs) in medical education.
Funder: Department of Health and Social Care

Abstract: Background: Accurate and fair assessment of medical students' clinical skills is vital to ensure competence. Current best practice for examining students' skills uses OSCEs (Objective Structured Clinical Examinations): students rotate around several standardised tasks (called stations), examined by different examiners. Several parallel circuits of the same exam are required to accommodate students. Problematic examiner variability occurs due to: a/. individual examiners; b/. between different circuits of the exam; c/. and is likely between different medical schools, meaning some students may pass who shouldn't and vice versa. Benefits of previous interventions have been limited. Comparing effects a-c is hindered by nesting: a lack of crossover between examiners in the performances they judge. No means exists to compare standards of judgement around the UK within a single exam.Aims: This research will develop, evaluate and implement a complex intervention in practice which aims to enable: robust comparison of OSCE circuit effects; potential comparison of individual examiner effects comparison of the standard of judgement between medical schools. Methods: The intervention will overcome nesting by requiring examiners to score common, station-specific, comparator videos. This will enable comparison of effects 1-3 and the potential to adjust students' scores to a fair score (the score they would receive with a completely neutral examiner). The intervention uses 3 sequential phases within real OSCE exams each time it is employed: Phase 1: a subset of students are videoed on each station within the OSCE Phase 2: examiners score common comparator videos, specific to the station they examine Phase 3: Many Facet Rasch Modelling (MFRM) is used to compare each of the 3 listed effects, producing fair scores Development: Study 1: Feasibility pilot. Volunteer students filmed live during their real OSCE exam. All examiners score live performances by students, then immediately score 2 additional station-specific common comparator videos. Up to 8 examiners examine each station in different circuits/ times. Analysis will consider dependability / data structure, producing initial effect estimates.Study 2: Understanding and optimising video representativeness. Will develop optimum filming approach to enhance representativeness of videos to live judgements, using action research. Iterative cycles of different filming approaches with comparison of examiners' judgements of the adequacy / similarity of judging live performances and videos, and students' views of acceptability of each method. Participants' comments collected and analysed thematically. Study 3: Determining sampling requirements and contribution of other influences. Similar to study 1, but examiners judge 4 comparator videos interspersed between live performances. Half of examiners judge performances in one order (A,B,C,D) and half in opposite order (D,C,B,A). Comparative influence of 4 effects determined: examiner stringency; contrast effects (bias due to examiners' recent experiences); longitudinal scoring changes; examiner idiosyncrasy. Model adjusted to more precisely model these effects. Sampling requirements (ideal number of comparator videos) determined.Evaluation: Study 4: Quantitative evaluation of intervention accuracy. Intervention used to determine fair scores for all student performances. Additionally, experienced examiners re-score videos of a randomly selected subset of performances, to give Group-Defined Accurate (GDA) score. Accuracy of model derived Fair Scores compared with GDA for all combinations.Study 5: Realist evaluation of implications of intervention. Theory-driven research to understand what mechanisms, in which contexts, lead to successful outcomes from the intervention. Focus on user perceptions (examiners, students, administrators, members of the public) and unintended consequences.Implementation: Study 6: Assessing transferability across institutions. Intervention collaboratively shared and employed across 4 institutions simultaneously during real exams. Comparison and adjustment of standard of examiners' judgements between medical schools.Workshops delivered to UK assessment academics on benefits and use of intervention; meetings with policy makers to promote adoption.

Lay abstract: Being able to accurately and fairly examine medical students on their clinical skills is vital to their education as well as to ensuring that they can practice safely in their future roles as doctors. Examining clinical skills relies on judgements by examiners: they observe students doing a clinical task (or a simulated version of the task) and judge the performance according to set criteria. Even when examiners are carefully trained, and the criteria are clearly specified, there can be a lot of variability between different examiners in their judgements of the performance. For example, in a previous study, different, experienced examiners gave scores of 3 out of 9 (a fail) and 9 out of 9 (an excellent pass) to the same student in the same videoed performance. This kind of variability is clearly a problem: it means that unsafe students could potentially pass; good students could unfairly fail; and the educational messages which students receive can be confusing.Students are usually examined in OSCE exams, where they rotate around a series of 10 or 12 different tasks (called stations), and at each station they are observed and judged by a different examiner. This helps because their performance is tested on a number of different skills and the different examiners balance each other out to some degree. Despite this, there is still substantial variability meaning we're not that confident in the final scores which students receive. This is particularly important for students who are near to passing or failing the exam.Because many students need to be examined, medical schools usually run multiple identical versions of the OSCE, in different circuits or different locations, or at different times of day. Usually this involves different groups of examiners on each different circuit of the OSCE. Recent research has shown that these different groups of examiners may judge differently, so that students' scores are affected by the circuit of the exam they're allocated to. On a national level this may be more important. Plans are in place for national exams for students, but so far there is no clear means to make sure that the standard of judgement is the same around the country. This proposal will try to address all of these problems: examiner variability; the influence of different circuits of the exam on students' scores; and variations between medical schools in the standard of judgement in OSCE exams.The proposal will use an innovative procedure to more precisely estimate the influence of each examiner, and potentially to adjust students' scores to a position that is as if they had met a completely neutral examiner on each station of their OSCE exam. It will also enable comparison, and potentially adjustment, between different circuits of the OSCE exam, or even between different medical schools. The procedure will use 3 processes. Firstly, a subset of students will be videoed on each different station in the exam, using unobtrusive, ceiling-based cameras. Secondly, all examiners will score a number of common videos of student performances during or immediately after the exam. They will do this in addition to their usual examining duties. Each examiner will score videos of the same station they have just examined 'live'. Having all examiners score the same videoed performances will produce crossover between examiners, allowing comparison of examiners in a way which is not currently possible. Thirdly, all of this information will be used in a statistical analysis to compare the effects of each examiner on students' scores.The research is a series of studies which will pilot the new procedure, developing and refining it, and then testing its major assumptions. It will then test the procedure thoroughly. It will test whether the adjusted scores which the procedure produces are accurate (or at least more accurate than they would usually get from an examiner). It will also explore how people (students, examiners, administrators, patients and the public) perceive the procedure; whether they trust the scores it produces; and whether there are unintended consequences to using it. As long as these findings are supportive, it will then work to share and implement the procedure, using workshops and meetings to explain the procedure and show its benefits, and collaborating with other medical schools to pilot its use across multiple medical schools at the same time. This should enable comparison of the standards of different examiners across different medical schools around the UK for the first time.

Research activity codes: 8.3 POLICY, ETHICS AND RESEARCH GOVERNANCE
Health categories: GENERIC HEALTH RELEVANCE