title: Longitudinal clinical assessment of undergraduate dental students: building an argument for its validity
Creators: name: Dickie, Jamie Daniel Robert
Publisher: name: University of Glasgow
Year: 2021

description: Background: Assessment of healthcare professionals plays a pivotal role in safeguarding patients by ensuring practitioners have been appropriately trained before being permitted onto professional registers. This prevents the public from being treated by those who are not fit to practise healthcare subjects, including dentistry. In the UK, dental schools must provide the General Dental Council (GDC) with evidence that students have attained the necessary educational outcomes and are suitable to join the professional register. The GDC delegates responsibility of choosing appropriate assessment methods to obtain such evidence to the dental schools themselves. As part of their undergraduate assessment repertoire, some UK dental schools have adopted longitudinal assessment methods to measure development and consistency of competent performance in clinical environments. Although these longitudinal methods create a rich database of multiple points of evaluation over the duration of the Bachelor of Dental Surgery (BDS) curriculum, there is currently little evidence to support their use for assessing development of clinical competence. Therefore, there is a need to conduct thorough analyses of longitudinal clinical data using robust statistical methods and create evidence to support their validity for this purpose. Aims: This thesis aims to investigate the content and criterion validity and reliability of longitudinal clinical assessment, which will contribute towards a validity argument on its use in assessing the development of clinical competence among undergraduate dental students. It will also explore how the evidence for validity could be used to enhance assessment within dental education. Research design: A mixed methods approach, with quantitative and qualitative approaches, was adopted to address the study aims. For the quantitative component, statistical escriptions, and group-based trajectory models (GBTMs) tracking individual undergraduate’s clinical performance over time were produced from longitudinal clinical assessment (LIFTUPP©) data for three dental student cohort’s (2017-19; n=234). Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Distinct trajectories were created using a performance indicator 5 as the threshold, which were then used to investigate the concurrent and predictive subtypes of criterion validity. Concurrent validity was investigated by linking and cross-tabulating LIFTUPP© trajectory group memberships with BDS examination performance (mean scores and a “top 20%” performance in each BDS year). Predictive validity was investigated by linking and cross-tabulating undergraduate LIFTUPP© trajectory group memberships with postgraduate clinical performance trajectory group memberships generated from Longitudinal Evaluations of Performance (LEPs). Reliability was calculated using Cronbach’s alpha. For the qualitative component of the study, a series of online focus groups with key stakeholders within dental education were conducted. Participants were presented with the results of the quantitative analyses and their opinions on how these data could be used to enhance assessment within dental education were canvassed. Transcripts of the focus group discussions were analysed using thematic analysis to identify themes (i.e., patterns) of interest within the data. Results: LIFTUPP© GBTMs with a threshold performance indicator of 4 resulted in all students following a single trajectory in all three cohorts and showed progressive development of clinical competence over three BDS clinical years, satisfying criteria for content validity. GBTMs with a threshold performance indicator of 5 provided at least two distinct trajectories of student clinical performance. According to the Bayesian information criterion (BIC), models with two distinct trajectories fitted the data best and a “better” performing trajectory was identifiable in each cohort. In the two most recent cohorts, students who were more likely to belong to the “better performing” LIFTUPP© trajectory scored higher (on average) in the undergraduate examinations for each BDS year. This association was not observed for cohort 1. Students allocated to “better performing” LIFTUPP© trajectories were more likely to also be assigned to “better performing” LEP trajectories in all three cohorts. Reliability for the undergraduate examinations was high in all three cohorts (&gt;0.88) and did not change substantially when longitudinal clinical assessment data were included. Comments from focus group participants appeared to provide further support for content validity. However, quantitative results were met with a degree of mistrust that seemed to stem primarily from previous experiences of operational issues associated with the LIFTUPP© assessment process and the absence of contextual data within the quantitative analyses. Conclusions: The upward trend of LIFTUPP© trajectory patterns suggested there is evidence that longitudinal clinical performance data have content validity for the assessment of clinical competence. Associations between better LIFTUPP© performance and better undergraduate examination outcomes and better postgraduate clinical performance in the two most recent cohorts were indicative of criterion validity. The lack of association in cohort 1 may have been due to poorer calibration among assessors following the initial adoption of LIFTUPP© into the BDS curriculum. Evidence for LIFTUPP© data reliability was inconclusive. This uncertainty may have resulted from using probabilities of student trajectory group membership as the metric for longitudinal clinical assessment in the calculation of Cronbach’s alpha. Therefore, further investigations on LIFTUPP© data reliability are required. Data processing procedures and suggestions from focus group participants revealed there is a need to improve current assessment practices and data collection to allow other investigations on validity to be pursued and to further increase confidence in the results produced by this study. Some data collection issues encountered in relation to LIFTUPP© and undergraduate examinations have since been resolved, meaning studies involving subsequent student cohorts should seek to incorporate LIFTUPP© communication, management and leadership, and professionalism data as well data from clinical case presentation examinations and one-off clinical competence tests. Overall, the study provides an early contribution towards a validity argument on the use of longitudinal clinical assessment in assessing development of competence in undergraduate dentists and provides a starting point from which consequent studies can be based. The study should now be expanded into different settings, e.g., other dental schools and disciplines (such as medicine, nursing, and veterinary medicine), to confirm and build upon these initial findings.