Towards an Automatic Evaluation of (In)coherence in Student Essays
Authors: Filippo Pellegrino, Jennifer Carmen Frey, Lorenzo Zanasi (EURAC, 2024)
Topic: Language Models

Abstract
Coherence modeling is an important task in natural language processing (NLP) with potential impact on other NLP tasks such as Natural Language Understanding or Automated Essay Scoring. Automatic approaches in coherence modeling aim to distinguish coherent from incoherent (often synthetically created) texts or to identify the correct continuation for a given sample of texts, as demonstrated for Italian in the DisCoTex task of EVALITA 2023. While early work on coherence modelling has focused on exploring definitions of the phenomenon, exploring the performance of neural models has dominated the field in recent years. However, coherence modelling can also offer interesting linguistic insights with pedagogical implications. In this article, we target coherence modeling for the Italian language in a strongly domain-specific scenario, i.e. education. We use a corpus of student essays collected to analyse students' text coherence in combination with data perturbation techniques to experiment with the effect of various linguistically informed features of incoherent writing on current coherence modelling strategies used in NLP. Our results show the capabilities of encoder models to capture features of (in)coherence in a domain-specific scenario discerning natural from artificially corrupted texts.

Introduction
Argumentative essay writing is a fundamental objective in education for both vocational schools and high schools in Italy, as indicated in AUTHOR. It requires students to present arguments supported by personal knowledge or external sources in a coherent and convincing manner. However, writing coherent texts poses both cognitive and linguistic challenges to novice writers and textual competences related to it are frequently claimed to be insufficient, putting pressure on the educational system.\\ Automatically discerning incoherent texts or passages could help teachers to better understand students' problems and give targeted instructions, while students would benefit from more frequent and more timely feedback. However, to date, most NLP research in automatic coherence modelling focused on semantic similarity between two parts of texts using mostly well-formed newspaper or Wikipedia texts, offering little information for educational contexts. \\ In this study, we explore coherence from an educational perspective, utilizing recent language models and data perturbation techniques to probe their value for linguistically informed and informative automatic coherence evaluation for student essays. While large language models have been used successfully in domain general coherence modelling before, we test their effectiveness for text analysis in this domain-specific scenario, taking into account both surface and non-standard language features. We discuss: \item data perturbation techniques to artificially reproduce real-life scenario incoherence in textual data \item a custom probing task design \item automatic evaluation of coherence using different encoding models The results of our experiments show the performances of encoder models in recognizing patterns of (in)coherence in a domain-specific educational context such as upper secondary school student essays. The paper is organized as follows: Section related_work provides an overview of previous approaches to coherence modelling and NLP data perturbation with a focus on Italian NLP. Section dataset introduces the data we used for this study, giving information on the research project it originates in as well as on the corpus design and annotation. Section methodology provides a detailed description of our methodology introducing our custom probing tasks (Section sec:Proping Tasks), used Models (Section subsubsec:models) and text encoding sec:encoding as well as a description of the two analyses performed (Section sec:performance and Section error). Sections results and discuss present and discuss our results and Section conclusions concludes the article with final considerations.

Conclusion
In this paper, we presented an evaluation of coherence modelling techniques for detecting incoherence in student essays based on surface-level features of incoherence. We used the ITACA corpus of Italian upper secondary school essays to perform a number of classification techniques using data perturbation and BERT-based text encoding methods. After a preliminary comparison between pre-trained and fine-tuned models we adopted the best performing one according to our results. The results of the chosen tasks are influenced by the implementation of the perturbation technique, the encoding ability of the model, and the amount and the quality of the data the model is pre-trained on. The best performances are bounded to the model pre-trained with the highest amount of data (GilBERTo). We based our evaluation on simple f1 measures considering this sufficiently indicative of the encoding ability of the model applied to each specific probing task.\\ Since we mainly tested custom perturbation techniques and the encoding abilities of BERT models, future research directions might involve data perturbation techniques enhancement, XAI techiques for model behaviour analysis AUTHOR and the exploitation of state-of-the-art generative one shot and few-shot models in a highly domain-specific scenario such as school essays writing. We thank Fondazione Bruno Kessler Trento for their support on the ITACA corpus and for allowing us to use their student essay dataset for fine-tuning. \clearpage \appendix