Automatic Assessment of English CEFR Levels Using BERT Embeddings
Authors: Veronica Juliana Schmalz, Alessio Brutti (Libera Università di Bolzano, Fondazione Bruno Kessler, KU Leuven, 2021)
Topic: Language Models

Abstract
The automatic assessment of language learners' competences represents an increasingly promising task thanks to recent developments in NLP and deep learning technologies. In this paper, we propose the use of neural models for classifying English written exams into one of the CEFR competence levels. We employ pre-trained BERT models which provide efficient and rapid language processing thanks to attention-based mechanisms and the capacity of capturing long-range sequence features. In particular, we investigate on augmenting the original learner's text with corrections provided by an automatic tool or by human evaluators. We consider different architectures where the texts and corrections are combined at an early stage, via concatenation before the BERT network, or as late fusion of the BERT embeddings. The proposed approach is evaluated on two public datasets: the EFCAMDAT and the CLC-FCE. The experimental results show that the proposed approach can predict the learner's competence level with very high accuracy, in particular when large labelled corpora are available. In addition, we observed that augmenting the input text with corrections provides further improvement.

Introduction
Finding a system which objectively evaluates language learners' competences is a daunting task. Several aspects need to be considered, including both subjective factors, like age, native language, cognitive capacities of the learner, and learning-related factors, for example the amount and type of received linguistic input <cit.>. Indeed, language competences are not holistic, but concern different domains, so that considering the mere formal correctness of learners' language has been shown not to represent a proper assessment procedure <cit.>. Moreover, human evaluators, despite having to adhere to a pre-defined scale and guidelines, such as the CEFR <cit.>, have proved to be biased <cit.> and inaccurate <cit.>. For these reasons, new language testing methods and tools have been developed. Current state-of-the-art models, such as Transformers, allow to process numerous and complex linguistic data efficiently and rapidly, by means of attention-based mechanisms and deep neural networks that capture the relevant features for the targeted task. However, the creation and access to necessary language examination resources including annotations and metadata appear to date limited. In this paper, we propose using BERT-base models to automatically assign CEFR levels to language learners' exams. Our aim is examining the possibility of providing the system with previously generated corrections, either by humans or automatically with a language checker. Additionally, we want to analyse the impact of the amount of data on the accuracy of the model in the classification of written exams taken from the English First Cambridge Open Language Database (EFCAMDAT) <cit.> and the Cambridge Learner Corpus for the First Certificate in English (CLC-FCE) <cit.>. In this way, a significant turning point could be made both in improving the functioning of these automatic systems and in the future collection of data from other languages.

Conclusion
In this paper we presented an alternative approach for assessing the competences of English language learners using pre-trained BERT-base models. We structured a multi-class classification task to map the BERT embeddings of written exams from the EFCAMDAT and CLC-FCE corpora to different levels of the CEFR scale. Alongside the students' original texts and the provided manual corrections, we automatically generated corrected versions with LanguageTool.  Thus, we conducted several experiments varying both the type and quantities of model input, as well as the model typologies. Our results proved that BERT-models remarkably succeed in classifying CEFR proficiency levels starting from original texts, especially with numerically significant data. Moreover, we observed that adding automatic and manual corrections can contribute to better results.