Improving medical education assessments with statistics: applying item response theory to general practice
Creators: name: Buljan, Monika
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 490501
subject: Applied statistics

description: This research is designed for medical practitioners, education specialists and analytical specialists. Although the context is Australian General Practice, many health specialties may benefit from these findings. Statistical methods are applied to forecast exam enrolment figures, calculate Item Response Theory (IRT) statistics for historical exams, explore several test-equating considerations to create a common scale, apply the IRT findings to all exams, explore simulations, provide solutions for consistent standards (pass-marks) and may lower the economic burden on educational operations/ administration (thus costs to candidates).  Recommendations are provided in the final chapter. Furthermore, this research encourages IRT and other measurement theory researchers to publish timely findings as demonstrated in this niche area: IRT applied to General Practice assessments in Australia. 

Item Response Theory (IRT) is a measurement theory based on statistical models that aim to accurately measure abilities, attitudes or other variables in a test or questionnaire. This industry research project is a new application of IRT in the Applied Knowledge Test (AKT) for the industry partner, Royal Australian College of General Practitioners (RACGP). The aim of the AKT is to classify candidates (test-takers; practicing doctors under supervision) into `safe`' and `not yet safe' for independent clinical practice in Australia. 

The rationale of this project is that Australian General Practice (GP) training places were expected to double from 600 places per year in 2010 to 1200 places by 2014. They rose to 1500 places by 2015. The RACGP assess the medical knowledge and skills of most GP trainees via Fellowship Exams, before the trainees enter the workforce (practice without supervision). To meet assessment demand (as well as education standards, timelines and economic objectives of a not-for-profit organisation), the RACGP commenced a major capacity planning project in 2013. Part of the capacity planning project included investigating an alternative assessment framework. 

The RACGP questioned when and how many enrolments to expect due to increased training places. Time Series Analysis and Exponential Smoothing methods on historical enrolment figures provided several models for evaluation. An external organisation to RACGP (General Practice Education and Training) provided data to adjust the RACGP point estimates of the best forecast model. This was the basis of the business proposal for the capacity planning project (thus this research into IRT). The forecasts were an invaluable resource in RACGP operations. An evaluation in the late stages of this project showed the forecasts were accurate within 1-3%.

Following the forecasts, IRT analysis continued. An alternative measurement theory to Classical Test Theory (CTT) is IRT; a method that can lead to smarter testing for general practitioners in Australia and other medical specialties. The term smarter is aligned with the definition of smart devices; making processes easier, more convenient and obtaining more with less resources/ effort. Applying IRT models to an exam produces two measurements of interest: an item (question) difficulty score and a person (candidate) ability score. Two IRT models, the one parameter and two-parameter logistic models (known as 1PL and 2PL respectively), were applied to three de-identified AKT exams, Test A, Test B and Test C. The investigation included model-fit (how well IRT fit the AKT data). Applying either a 1PL or 2PL IRT model to future RACGP Applied Knowledge Tests is recommended.

Educators are interested in comparing results over time. Thus, some equating methods (pairwise and concurrent methods) were compared to map each exam on the same scale. Pairwise equating showed that the periods 2008.1 and before were potentially problematic for equating due to few link items. A Link Analysis (common items between exam administrations) was applied. One expects that a single item will perform similarly between two similar large groups of test takers. Consistent link items are ideal; have stable statistics over time. Poor link items may need to be considered as non-link items (that is, removed from the Link Set). Concurrent equating was applied to 2008.2 to 2015.1 AKT and compared to the pairwise data. Medical Educators (experts in GP and education) could not identify ideal or poor link items based on their medical expertise. However, statistical tools could rank link items from ideal to poor. 

 Once a common scale was established, analysis of historical standards (pass-marks or cut-scores) were retrospectively mapped which showed that standards changed over time. Applying IRT would ensure standards are always constant, thus making the exams consistent (fairer) for candidates (reliability). As for validity, the Item Bank, the storage database for items, was also analysed by comparing the tag metadata such as curriculum, clinical areas and demographics with everyday clinical practice and how it may lead to Computer Adaptive Testing (CAT). The items were tagged (classified) poorly and medical educators were required to rectify the item bank. Further validity is possible via the proposed pilot study of an IRT AKT. Finally, some simulations show what happens to the test length by changing other variables, namely in test reliability and root mean square error (RMSE). 

The thesis concludes that IRT is appropriate for RACGP and that care should be taken at each decision step. IRT is slowly gaining popularity in the Australian medical education sector but it has not been applied and published for GP assessment data at the vocational level of training (pre-workforce level). The model detail, assumptions, equating methods and other IRT-related properties have not been published for public access by any Australian vocational Medical College (like the RACGP), even though similar organisations use IRT for various medical assessments. Timely publication of these operational and theoretical considerations would help drive quality across all medical examinations in Australia and may ultimately lead to better patient outcomes.