Automated Short Answer Grading: A Simple Solution for a Difficult Task
Authors: Stefano Menini, Sara Tonelli, Giovanni De Gasperis, Pierpaolo Vittorini (Fondazione Bruno Kessler, Università dell'Aquila, 2019)
Topic: Language Models

Abstract
The task of short answer grading is aimed at assessing the outcome of an exam by automatically analysing students’ answers in natural language and deciding whether they should pass or fail the exam. In this paper, we tackle this task training an SVM classifier on real data taken from a University statistics exam, showing that simple concatenated sentence embeddings used as features yield results around 0.90 F1, and that adding more complex distance-based features lead only to a slight improvement. We also release the dataset, that to our knowledge is the first freely available dataset of this kind in Italian.

Introduction
Human grading of open ended questions is a tedious and error-prone task, a problem that has become particularly pressing when such an assessment involves a large number of students, like in an Academic setting. One possible solution to this problem is to automate the grading process, so that it can facilitate teachers in the correction and enable students to receive immediate feedback. Research on this task has been active since the ’60s (Page, 1966), and several computational methods have been proposed to automatically grade different types of texts, from longer essays to short text answers. The advantages of this kind of automatic assessment do not concern only the limited time and effort required to grade tests compared with a manual assessment, but include also the reduction of mistakes and bias introduced by humans, as well as a better formalization of assessment criteria. In this paper, we focus on tests comprising short answers to natural language questions, proposing a novel approach to binary automatic short answer grading (ASAG). This has proven particularly challenging because an understanding of natural language is required, without having much textual context, while grading multiple-choice questions can be straightforwardly assessed, given that there is only one possible correct response to each question. Furthermore, the tests considered in this paper are taken from real exams on statistical analyses, with low variability, a limited vocabulary and therefore little lexical difference between correct and wrong answers. The contribution of this paper is two-fold: we create and release a dataset for short-answer grading containing real examples, which can be freely downloaded at . Besides, we propose a simple approach that, making use only of concatenated sentence embeddings and an SVM classifier, achieves up to 0.90 F1 after parameter tuning.

Conclusion
In this paper, we have presented a novel dataset for short answer grading taken from a real statistics exam, which we make freely available. To our knowledge, this is the first dataset of this kind. We also introduce a simple approach based on sentence embeddings to automatically identify which answers are correct or not, which is easy to replicate and not computationally intensive. In the future, the work could be extended in several directions. First of all, it would be interesting to use deep-learning approaches instead of SVM, but for that more training data are needed. These could be collected in the upcoming exam sessions at University of L’Aquila. Another refinement of this work would be to grade the tests by assigning a numerical score instead of a pass/fail judgment. Since such scores are already included in the released dataset (the degrees), this would be quite straightforward to achieve. Finally, we plan to test the classifier by integrating it in an online evaluation tool, through which students can submit their tests and the trainer can run an automatic pass/fail assignment.