Quale testo è scritto meglio? A Study on Italian Native Speakers’ Perception of Writing Quality
Authors: Aldo Cerulli, Dominique Brunato, Felice Dell'Orletta (Università di Pisa, CNR-ILC, 2021)
Topic: Learner Corpora and Language Acquisition

Abstract
This paper presents a pilot study focused on Italian native speakers' perception of writing quality. A group of native speakers expressed their preferences on 100 pairs of essays extracted from an Italian corpus of compositions written by L1 students of lower secondary school. Analysing their answers, it was possible to identify a set of linguistic features characterizing essays perceived as well written and to assess the impact of students errors on the perception of text quality. The paper describes the crowdsourcing technique to collect data as well as the linguistic analysis and results.

Introduction
The institution of distance learning paradigms, which has become crucial during the Covid-19 pandemic, showed the need to provide schools and universities with Natural Language Processing (NLP)-based tools to assist students, teachers and professors. Nowadays, language technologies are more and more exploited to develop educational applications, such as \emph{Intelligent Computer-Assisted Language Learning (ICALL)} systems \cite{granger-2003} and tools for automated essay scoring \cite{attali-burstein-2006} or automatic error detection and correction \cite{ng-2013}. A fundamental requirement for developing this kind of applications is the availability of electronically accessible corpora of learners' productions. Corpora created so far differ in many respects. For instance, considering the types of examined learners, they can gather productions written by L2 students or by native speakers: the former have been built for many languages (e.g. English, Arabic, German, Hungarian, Basque, Czech, Italian), while the latter are mainly available for English. In both cases, a peculiarity of existing corpora is that they are cross-sectional rather than longitudinal. A notable exception in the context of Italian as L1 -- which is the focus of our contribution -- is represented by \emph{CItA (Corpus Italiano di Apprendenti L1)}, which was jointly developed by the Institute for Computational Linguistics of the Italian National Research Council (CNR) of Pisa and the Department of Social and Developmental Psychology at Sapienza University of Rome \cite{barbagli-2016}: it is the first digitalized collection of essays written by the same group of Italian L1 learners in the first two years of the lower secondary school\footnote{The corpus is freely available for research goals at \href{http://www.italianlp.it/resources/cita-corpus-italiano-di-apprendenti-l1/}{http://www.italianlp.it/resources/cita-corpus-italiano-di-apprendenti-l1/}}. The diachronic and longitudinal nature of \emph{CItA} makes it particularly suitable to study the evolution of L1 writing competence over the two years, assuming that many remarkable changes in writing skills occur in this period. For instance, in their recent work, \citet{miaschi-2021} showed that it is possible to automatically learn the writing development curve of students: they extracted a wide set of linguistic features from the essays and used them to train a binary classification algorithm able to predict the chronological order of two productions written by the same pupil at different times. The present study ranks among research based on \emph{CItA}, but chooses a different approach from the one just mentioned: instead of tracking the development of students' writing competence, we focused on the perception of writing quality by Italian L1 speakers with the aim of understanding whether it is possible to find the linguistic features that are crucially involved in the distinction between `better' and `worse' essays according to our target reader.\paragraph{Contributions} To the best of our knowledge, this is the first paper that (i) introduces a dataset of evaluated essays in terms of perceived writing quality by means of a crowdsourcing task, (ii) deals with the correlation between linguistic features and perceived quality of writing and (iii) assesses the impact of students errors on quality perception. 

Conclusion
We presented a pilot study towards the identification of the linguistic features that are own of well written perceived essays. We collected Italian natives' preferences on 100 pairs of essays written by L1 students, that we analysed in terms of linguistic profiling and errors distribution. Our results reveal an interesting correspondence between annotators' judging criteria and writing instructions that L1 learners receive by teachers. Our findings could be interpreted as an indicator of the reliability of our data and, more in general, could suggest the effectiveness of crowdsourcing methods to quickly build large and reliable datasets. Considering the lack of Italian corpora of graded essays, such datasets could be valuable resources for the development of Computer-Assisted Learning Systems. The limited size of our dataset certainly reduced the amount of results. Thus, we have to expand it (i) by collecting more annotations for the already existing surveys and (ii) by creating and distributing new surveys in order to gather judgments on new pairs of essays. Analysis on the enlarged dataset could provide more features that are own of good essays. Following the model of <cit.>, we could use the results to train a classifier that, given a pair of essays, recognizes the best written one. The tool would not presume to replace teachers, but it could be a valuable teaching aid. Students could use it to get an immediate and preliminary self-assessment on their written productions so as to better understand their mistakes and hopefully avoid repeating them. Such tools can be very useful if integrated into educational processes based on distance learning paradigms, which need adequate technological infrastructures to be really efficient.