Confronto tra diversi tipi di valutazione del miglioramento della chiarezza di testi amministrativi in lingua italiana
Authors: Mariachiara Pascucci, Mirko Tavosanis (Università di Pisa, 2024)
Topic: Text Simplification

Abstract
The paper presents a comparison of different types of evaluation of administrative texts in the Italian language on which a Clarity improvement intervention was carried out. The Clarity Improvement was performed by human experts and ChatGPT. The evaluation was carried out in four different ways: by expert evaluators, used as a reference; by evaluators with good skills, subject to dedicated training; by generic evaluators recruited through a crowdsourcing platform; by ChatGPT. The results show that the closest match to the results of the evaluation by expert evaluators was achieved, by a wide margin, by evaluators with good skills and dedicated training; the second best approach was obtained by requesting evaluation from ChatGPT; the worst approach was achieved by generic evaluators recruited through a crowdsourcing platform. Task features that may have influenced the outcome are also discussed.

Introduction
The spread of generative artificial intelligence systems has led to a large demand for evaluation of their capabilities. The type of assessment universally considered the most valid remains in general that made by human beings, but in practice it can be conducted in different ways and with very different results of value. Moreover, for some capacities there are still no shared scoreboards. There is no doubt that the latter category also includes an assessment of the overall improvement in the clarity of the texts in the Italian language, which is the subject of the analysis described here. The existing objective indices for the analysis of texts, such as GULPEASE or the quantification of the words included in the Basic Vocabulary, actually describe only limited aspects of any text. For clarity in itself, while the indications on how to write clearly abound (an updated summary is given in [1]), broad consensus criteria for product evaluation have never been codified [2]. Of course, many current evaluation methods provide at least a first orientation in most cases. For example, [3] has shown that through crowdsourcing it is possible to obtain a generic but reliable indication on the improvement of the clarity of English texts. However, studies on the effectiveness of such practices are still very few and there is no doubt a great need to improve the current level of knowledge. This contribution is part of this context as it compares different methods of assessing the improvement of the clarity of texts. The subject matter of the evaluation was rather broad texts, representative of the administrative Italian and made clearer through human intervention and reformulation with ChatGPT (version 3.5); the context, which saw the implementation of several related evaluation activities, is described in detail in [4]. For the purpose of this contribution, the evaluation was carried out in four different ways: by expert evaluators, used as reference; by well-trained evaluators, the subject of dedicated training; by generic evaluators recruited through a crowdsourcing platform; by ChatGPT. In all cases, the same set of assessment indications was used. The results were analysed in [4] for the information they provide regarding the ability of systems such as ChatGPT to effectively improve text clarity. More specifically, the difference in judgements in relation to the four methods of assessment will be shown here.

Conclusion
The results of the different valuation methods could at first glance be interpreted as a devaluation of crowdsourcing, compared to which the simple request to ChatGPT is able to deliver higher quality results. However, it is clear that the characteristics of the activity carried out make it advisable not to draw too general conclusions. First of all, it calls for caution that evaluation depends on the scale used. In a context where it is known that the vote can only be 4 or 5, at the end of the day, the simple random allocation of the score would give 4.5 to both Group A and Group B, moving away from the expert judgement with 0.26 for the overall assessment and 0.34 for aspects 1, 2 and 5, values very close to those provided by the trained evaluators. In these circumstances, it seems useful first of all to create more specific and targeted evaluation grids. The high performance of the current systems, moreover, certainly makes less useful than in the past scales 1-5 in which score 1 must be assigned to a completely incomprehensible text and score 5 to a perfectly understandable text. Certain limits of the analysis should also be taken into account. One of these is the involvement of the authors in the rewriting of some texts: even if the characteristics of the evaluation make in our opinion very limited the risk of alterations, it is expected to modify the protocol for future activities of the same kind, delegation all the rewritings to third parties. For the evaluation of ChatGPT-generated texts it may also be useful to have the texts evaluated by a different system and, in general, to extend and repeat the evaluations is of course necessary to validate the results. However, the results certainly call for attention to be paid to the limitations of today's widespread practices such as crowdsourcing, which on the task under consideration have shown a considerable deviation from the evaluation of experts. Moreover, if the rapid and economic evaluation provided by systems such as ChatGPT were to be regularly confirmed as closer to expert evaluation than crowdsourcing, the motivations for crowdsourcing itself would disappear.