Title: Enhancing selection of the healthcare workforce: a programme of psychometric epidemiology
Funder: Department of Health and Social Care

Abstract: Effective medical treatments will not translate into improved patient outcomes unless delivered by a competent, professional and compassionate healthcare workforce. Nevertheless, at present there are wide variations in the approaches used for both medical selection and for NHS staff recruitment in general. Ideally, selection should result in the appointment of candidates most suited to an occupational role. For example, those with caring roles should demonstrate empathy and integrity. Thus there is a need to establish an evidence base for selection methods if a professional and competent workforce is to be maintained. Academic performance, especially early in undergraduate training, may be predicted by prior educational attainment and aptitude' test scores, evaluating cognitive ability. However, little is known about which factors are likely to predict later performance in doctors, though there may be some early indicators regarding which individuals may be at risk of future disciplinary action. Moreover, the factors that tend to predict early performance in medical students appear less relevant as training progresses, and vice-versa. Additionally, the issues identified in the Francis report (2013) and the abuses recorded at Winterbourne View have led to the requirement for 'Values Based Recruitment' (VBR) in the NHS; from March 2015 all healthcare providers are expected to implement VBR. Thus, there is an urgent need to develop robust approaches to the selection of healthcare staff on the basis of personal attributes. However, the most cost-effective approach to VBR is currently unknown. The use of Situational Judgment Tests (SJTs), where knowledge of professional behaviour is evaluated via a candidate's responses to presented social situations, offers a plausible, practical and legally defensible method. Early, emerging evidence regarding the validity of SJTs for medical selection suggests that the scores may predict performance in inter-personal and clinically orientated tasks. However, research into their psychometric properties is urgently required in order to optimise the way the tests are constructed and used as a component of healthcare recruitment. Moreover, emerging large scale quantitative data relating to selection and training in doctors provides unprecedented opportunities to explore the educational, demographic and cognitive predictors of later performance in medical practitioners. This programme of work consists of two work streams: Work addressing the question of which factors in medical students predict later performance in qualified doctors?'. A taxonomy of potential outcomes of interest will be developed. Advanced statistical modelling techniques will be applied to routinely arising data created during the medical selection and training processes to link predictors with key outcomes. These will include existing SJT response data. Data will be accessed via collaborators and the emerging UK Medical Education Database (UKMED).Work exploring the question 'can SJTs can be used as an effective component of VBR in the NHS?'. This will be achieved by the development, validation and implementation of SJTs within the appointments process of health workers in a mental health setting. Individual performance on SJTs will be linked to a conscientiousness index' (an objective metric of behaviour), performance in a high-fidelity simulation, team climate and performance in a large mental health trust, in order to explore links with patient outcomes and perceptions. Thus, lessons learned from medical selection will be tested more widely. This fellowship is therefore sought to train and develop an independent clinical researcher, based in one of the UK's top centres of medical education, to lead a programme of research on staff selection. This will lead to consistent, evidence-based, approaches to this issue. The selection of staff with higher levels of inter-personal funtioning and professionalism will result in improved care and reduce the risks of patients being exposed to poor practice.

Lay abstract: Effective treatments in medicine will not translate into better patient outcomes unless delivered by a conscientious, competent, compassionate and professional workforce. Consequently, selecting tomorrow's doctors is a high stakes issue. At present there is wide variation in how medical schools select entrants. Academic performance in the early years of medical school may be predicted by prior educational attainment and aptitude' test scores, which evaluate intellectual ability. However, little is known about which factors predict later real world' performance in qualified doctors, though there may be some early indicators regarding which individuals may be at risk of future disciplinary action. It is not currently clear which outcomes are the most important to predict; some may be more closely associated with patient care than others (e.g. Fitness to Practise issues versus written exam performance). Thus, it is important to develop an evidence base for existing selection strategies that takes into account the differing nature of outcomes.Recently there has been an interest in testing non-academic attributes when choosing future doctors. These may predict professionalism and include such features as empathy and integrity. Failure to select on these factors may have serious consequences for patients; several scandals have highlighted the importance of compassion and conscientiousness in healthcare. These include the issues relating to poor care at Staffordshire Hospital (Francis report, 2013) and the abuses of disabled individuals at Winterbourne View care home. These situations have led to the requirement for 'Values-Based Recruitment' (VBR) in the NHS. VBR involves attempting to evaluate the extent to which an individual holds, and exhibits, the values in the NHS constitution (working together for patients, respect and dignity, commitment to quality of care, compassion, improving lives, and everyone counts). There are a number of ways to test for relevant attributes (e.g. at interviews). However, there is emerging evidence from occupational psychology and initial findings in qualified doctors, that suggest Situational Judgment Tests (SJTs) may offer an efficient way of testing for traits associated with professionalism. SJTs evaluate knowledge of professional behaviour via a candidate's responses to presented social situations. Though promising, at this point it is critical to establish what role SJTs should play in the selection of doctors, and, more widely in health service recruitment. It may be that SJTs can be used to identify and exclude candidates that are at risk of unacceptable workplace behaviours. Moreover, SJTs may be less prone to ethnic bias when compared to tests of intellectual ability. Thus, the effective implementation of SJTs to NHS recruitment has implications for patient care and safety as well as the widening of access to the health professions. This proposed programme of work consists of two inter-related streams. Firstly, we ask what are the early key predictors of later work related performance in doctors?'. This will involve collaborating with stakeholders in order to link and analyse the increasing quantities of routinely collected data generated from medical selection and training. A framework for classifying the outcomes of interest will be developed. Using advanced statistical methods the ability of a range of factors at application to predict the key outcomes most relevant to patient care (e.g. Prescribing Safety Assessment scores, Fitness to Practise declarations etc), will be evaluated. This will allow for the production of clear guidance on the implications of the current selection criteria. SJT responses in both medical applicants and qualified doctors will also be analysed. These latter findings will feed into the second workstream, which asks can SJTs be used effectively as a component of VBR in the NHS?'. This will involve the development of a pool of SJT items that will be piloted in a group of mental health workers from different professional disciplines, as proof of concept. A recruitment toolkit, including SJTs, for health employers will be developed and the validity and feasibility of the system evaluated. Thus, the intention is to take the understanding gained from analysis of SJT data in medical selection and use to it to develop a system that has the potential to be adapted for use across other healthcare professions. This fellowship is intended to support the development of a clinical researcher, based in one of the UK's top centres of medical education, to develop and lead a programme of research on staff selection that will contribute to an NHS staffed by a professional and conscientious workforce. This will result in improved care and reduce the risks to patients of staff malpractice.

Research activity codes: 8.1 ORGANISATION AND DELIVERY OF SERVICES
Health categories: GENERIC HEALTH RELEVANCE