You write like a GPT
Authors: Andrea Esuli, Fabrizio Falchi, Marco Malvaldi, Giovanni Puccetti (CNR-ISTI, Professional writer and independent researcher, 2024)
Topic: Textual Genres & Literature Linguistics

Abstract
We investigate how Raymond Queneau's Exercises in Style are evaluated by automatic methods for detection of artificially-generated text. We work with the Queneau's original French version, and the Italian translation by Umberto Eco. We start by comparing how various methods for the detection of automatically generated text, also using different large language models, evaluate the different styles in the opera. We then link this automatic evaluation to distinct characteristic related to content and structure of the various styles. This work is an initial attempt at exploring how methods for the detection of artificially-generated text can find application as tools to evaluate the qualities and characteristics of human writing, to support better writing in terms of originality, informativeness, clarity.

Introduction
The extraordinary writing ability of the latest chatbots and virtual assistants based on Large Language Models (LLMs) poses a significant question for anyone who attempts to write today —- be they a scientist, a writer, or a lover: is it worth the effort to engage in the act of writing? For those not hindered by excessive laziness and who, with courage, still tackle writing with determination and passion, this question implies a more specific one: am I writing a text that an artificial intelligence could not have produced? We believe that the answer to this question may, in the future, come from the LLMs themselves given that they are designed to assess the probability of the occurrence of the next word in a text. We envision a future where LLMs, although widely used to produce essentially obvious texts, will assist those who still engage in writing to create texts worth reading, if only because the artificial intelligence, having read and statistically evaluated almost everything ever written, considers them non-obvious and distinct from what it would have produced itself. The ability of LLMs to evaluate the probability of the next word in a text stems from the extensive corpus of writing they are trained on. Consequently, their evaluation of a piece of writing is ultimately based on an indirect comparison between the given text and the entire body of literature they have been exposed to. Using LLMs to assess how much a text differs from the production capabilities of LLMs inherently implies an evaluation of the novelty it represents compared to known literature. Starting to move in this direction, this article explores whether an LLM can be used to help humans answer this question. In this first attempt we do this not based on the content intended for communication but on the style. We have conducted a preliminary study on the possibility of using LLMs to evaluate how and to what extent a certain writing style and/or a specific text differs from what a machine can achieve. We took as a reference Raymond Queneau's ``Exercises in Style'' AUTHOR, which draws from Erasmus of Rotterdam's ``De Utraque Verborum ac Rerum Copia'' AUTHOR a bestseller widely used for teaching how to rewrite pre-existing texts and how to incorporate them into a new composition. In Queneau's work, the same simple story is revisited each time in a different literary style. We asked ourselves and conducted experiments on how much the texts in various styles used by Queneau differ from the writing abilities of LLMs, which have acquired their skills by learning statistical relationships from vast amounts of text. Calvino had already attempted to answer this question: ``What would be the style of a literary automaton?'' He replied, ``The test for a poetic-electronic machine will be the production of traditional works, of poems with closed metric forms, of novels with all the rules''. We believe it has indeed happened this way, as today's chatbots and virtual assistants are built from a language model. In this work, we provide initial evidence that language models recognize those texts that are more traditional, particularly used in spoken language or by classical characters as more probable while they deem more unlikely experimental and innovative texts. However, we find evidence that even for powerful LLMs it remains difficult to cut a clear line between experimental texts and those that instead incur the risk of becoming unreadable.

Conclusion
This work is a first exploration of the idea of designing tools that evaluate how and to what extent a writing style and/or a specific text differs from what a machine can achieve. We tested for this task the use machine generated text detection tools, under the hypothesis of a correlation between their detection scores and our goal of discovering the many facets that build an original human written text. We applied them to Queneau's exercises in style, in which the same story is written using a rich and varied set of writing styles. We have found a consistent correlation between the scores assigned by detection methods, across detection methods and across languages. The comparison of the styles with their detection scores indicates that lower scores from detection methods are correlated with the use of unusual terms or syntax, while higher scores are more related to styles that are based on a clean and more prose, with a smooth transition among this two extremes. The ranks thus do not indicate a ``better'' or a more ``interesting'' style, yet they confirm Calvino's statement we reported in the introduction: content that is akin to a machine-generated one is the one that produce ``traditional'' content, following the main rules of writing. Writers willing to depart from sounding ``ordinary'' could indeed use detection methods to estimate these aspects on their content, with the caveat that while a mid-level detection score may suggest some original traits in text, low scores may not indicate a more original or interesting text, but they may likely derive from an obscure or plainly unreadable text. Given the positive results of this first investigation, future developments will be based on the use of texts specifically written for this activity. This will have the advantage of having full control over the contents and to have the guarantee that they have never been part of the LLMs training data. \FloatBarrier This work was partially supported by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - "FAIR - Future Artificial Intelligence Research" - Spoke 1 "Human-centered AI", funded by European Union - NextGenerationEU.