Variability and agreement in assessor responses to undergraduate essays: an issue for quality assessment in higher education?

With improved access to higher education (HE) and growth in student mobility, student cohorts have become increasingly diverse in terms of educational, cultural and linguistic background. In view of this changing student population, this thesis considers some of the challenges faced by teachers of university disciplines in their role as assessors of student work. Recent literature on assessment in HE has drawn attention to the lack of research on assessment practices, which remain, on the whole, poorly described and not well understood by practitioners, students, and researchers alike. Given widespread criticisms of the quality of assessment in HE (standards, reliability, and validity), this is an issue of some concern. Moreover, the lack of research in this area is surprising, not only in view of contemporary understandings of the central role of assessment in student learning, but also because of the growing demand for accountability and transparency in HE assessment. In light of this deficit in our knowledge of assessment practices, this thesis considers the questions of if, and how, assessors’ responses to university essay assignments are influenced by student language background. To this end, a study was carried out at an Australian university to investigate the impact of English native-speaker (NS) or nonnative-speaker (NNS) authorship on assessors’ responses to undergraduate essays. Ten assessors marked five NS and five NNS essays, and data were collected from the following sources: i) assessors’ concurrent ‘think aloud’ verbal reports; ii) essay grades; iii) assessors’ retrospective verbal and written reports; iv) essay texts; and v) interviews with assessors. The findings relate to two dimensions of the essay marking process: the performance features that assessors focussed on; and the grades awarded to the essays. Firstly, differences were observed between responses to NS and NNS essays in both the criteria for assessment, and the grades awarded. These differences relate primarily to the relative extent of attention given to matters of content (greater for the NS essays) compared with grammar (greater for the NNS essays). Secondly, a striking degree of variability was found amongst assessors in the grades they awarded to the essays, with many essays receiving the whole range of possible grades. This variability derived from differences not in the criteria used for assessment, but in assessors’ interpretations of these criteria, with matters of essay content being most subject to differing interpretations. As a result, owing to the greater focus on the content of the NS essays, inter-assessor grading was less reliable for the NS essays compared with the NNS essays. The implications of these findings are discussed in relation to the ways in which assessment practices may impinge on equitable and valid assessment of student performance. The thesis concludes by considering how potential improvements to assessment practices could take into account the challenges involved in catering to larger and more diverse student populations.