The impact of between subject differences in grading standards on composite measures of student ability in the UK

Composite scores are created in many education systems to combine the results from multiple assessments into a single measure of student (or school) achievement. In England, two composite scores are used as key school performance measures, Progress 8 (P8) and Attainment 8 (A8). Both of these scores make the implicit assumption that grades received by candidates in different subjects are interchangeable in terms of their relative grading standards or ‘difficulty’. No method is currently employed to scale or equate scores across different subjects. This has led to concern that schools can be penalised by these measures by entering students for ‘more difficult’ subjects. This perceived ‘perverse incentive’ is potentially driving unwanted behaviour by causing schools to not enter students for certain subjects, most noticeably modern foreign languages (MFL).
This study aimed to evaluate if schools were being unfairly penalised by choosing to enter students to MFL subjects through the methodology currently used to calculate A8 and P8. A secondary aim was to evaluate if this bias could be effectively removed by equating subject scores using a partial credit model.
Analysis showed that using common statistical methods MFL subjects appear among the most ‘difficult’ at GCSE in terms of statistical grading standards. Utilising a simple score adjustment and a more complex simulation showed that this difference in difficulty could potentially lead to a small negative bias in performance scores for schools entering a large number of subjects to MFL subjects. Although a small effect, this bias could still have a meaningful effect on a schools relative ranking and performance banding. Imputation of student scores indicated that schools could potentially benefit by entering students for alternative subjects instead of MFL, as long as they could be included in the English Baccalaureate ‘bucket’ of performance measures but not if it was replaced by any other subject. However, despite this estimate negative bias, when real school results were modelled, entering students for MFL subjects appeared to have a positive impact on A8 and P8 scores. This may be due to schools compensating for the existing bias by entering students for MFL with a higher subject specific aptitude, potentially masking any negative effects in the empirical data. Trialling the application of a partial credit model suggested it could be used to effectively compensate for differences in subject grading standards. However, it would add significant complexity to the current method and would potentially introduce additional issues and complications.
It is concluded that the current methodology for calculating A8 and P8 may be driving unwanted behaviour in the system, potentially working against one of the key aims of the government to promote a “broad and balanced curriculum”. However, as the performance measures are utilised both for school accountability and parental choice through ‘school league tables’, it is unclear which of these should be prioritised and therefore whether introducing a significantly more complex method would be beneficial. Other research suggests there are more pressing issues with these performance measures as they take no account of student background characteristics causing more serious systematic bias.

Keywords: MEA 2020
MEA