AI ‘News’ Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italian (2024)

Abstract: Large Language Models (LLMs) are increasingly used as ‘content farm’ models (CFMs), to generate synthetic text that could pass for real news articles. This is already happening even for languages that do not have high-quality monolingual LLMs. We show that fine-tuning Llama (v1), mostly trained on English, on as little as 40K Italian news articles, is sufficient for producing news-like texts that native speakers of Italian struggle to identify as synthetic. We investigate three LLMs and three methods of detecting synthetic texts (log-likelihood, DetectGPT, and supervised classification), finding that they all perform better than human raters, but they are all impractical in the real world (requiring either access to token likelihood information or a large dataset of CFM texts). We also explore the possibility of creating a proxy CFM: an LLM fine-tuned on a similar dataset to one used by the real ‘content farm’. We find that even a small amount of fine-tuning data suffices for creating a successful detector, but we need to know which base LLM is used, which is a major challenge. Our results suggest that there are currently no practical methods for detecting synthetic newslike texts ‘in the wild’, while generating them is too easy. We highlight the urgency of more NLP research on this problem.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15312–15338 August 11-16, 2024 ©2024 Association for Computational Linguistics
We investigate three LLMs and three methods of detecting synthetic texts (log-likelihood, DetectGPT, and supervised classification), finding that they all perform better than human raters, but they are all impractical in the real world (requiring either access to token likelihood information or a large dataset of CFM texts). We also explore the possibility of creating a proxy CFM: an LLM fine-tuned on a similar dataset to one used by the real ‘content farm’. We find that even a small amount of fine-tuning data suffices for creating a successful detector, but we need to know which base LLM is used, which is a major challenge.
Our results suggest that there are currently no practical methods for detecting synthetic newslike texts ‘in the wild’, while generating them is too easy. We highlight the urgency of more NLP research on this problem.

heading: 1 Introduction
text: The modern Large Language Models (LLMs) can generate increasingly fluent and plausible-sounding texts, which sparks concerns about their potential misuse by bad actors. One of the emerging problems is AI-driven news “content farms”: news-like sites filled with synthetic texts that are not necessarily serving a misinformation campaign, but are plausible-looking enough to deceive the readers and generate web traffic. Already in May 2023 NewsGuard reported that they found 49 such ‘outlets’ (Sadeghi and Arvanitis, 2023), and on June 4th
2024 their count1 was 840. Sometimes such sites publish original ‘content’, and sometimes they automatically ‘rewrite’ articles from real news outlets without attribution (Brewster et al., 2023). They are primarily created for serving programmatic ads, and even major brands may unwittingly support such ‘outlets’ (Ryan-Mosley, 2023). They already operate in many languages.2 The problem will likely only get worse with time, and it needs more attention both on the policy & regulation side and from the NLP researchers.
We illustrate how easy it is to create ‘content farm’ models (CFMs), and how hard to detect them, by considering a case that should be relatively tricky. In §3 we successfully turn Llama, a relatively old LLM mostly trained on English, into an Italian news CFM – by fine-tuning it on only 40k Italian news texts. For Llama65B this turns out to be sufficient to mislead native speakers of Italian, who identify synthetic texts with only 64% accuracy, vs 50% random guess (see §4). We also find that existing detection methods perform better
1https://www.newsguardtech.com/ special-reports/ai-tracking-center/
2NewsGuard AI Tracking Center currently states that they found such ‘outlets’ in Arabic, Chinese, Czech, Dutch, English, French, German, Indonesian, Italian, Korean, Portuguese, Russian, Spanish, Tagalog, Thai, and Turkish.
15312
than humans (§5), but are impractical in the real world (§7).
In §6 we explore an alternative approach: finetuning another LLM as a proxy for the real CFM and relying on its token likelihood scores as proxies for the scores of the real CFM. We find that this works with even a small amount of fine-tuning data (only 3% of the full fine-tuning dataset) but only given that we know which base LLM was used as CFM. We also experiment with using these proxy scores to identify the base LLM, but this method would also be impractical if there are dozens, if not hundreds possible alternatives.
We hope that our findings will boost similar investigations for other languages, and highlight the urgency of developing model-agnostic methods for synthetic text detection. To facilitate future research on Italian, we release (i) 15k news passages generated by models fine-tuned on the CHANGE-it dataset, an Italian news dataset (Mattei et al., 2020), (ii) ratings produced by 5 human annotators on 400 texts, with a balanced distribution of 50% humanwritten and 50% synthetic passages, and (iii) 600k synthetic alterations of both the original samples from the CHANGE-it dataset and the synthetic texts. The code and data are publicly available.3
We will not publicly release our fine-tuned LLMs (since they are best suited to be used as CFMs), but we welcome direct requests from researchers working on this problem.

heading: 2 Related work
text: Driven by the increasing number of strong openly available LLMs (Touvron et al., 2023a; Brown et al., 2020; Raffel et al., 2020; Jiang et al., 2023), several studies focused on the detection of synthetic text detection.
Ghosal et al. (2023) identified two main groups of approaches: those based on token likelihood, and supervised classification. Among the former, Mitchell et al. (2023); Su et al. (2023); Hans et al. (2024); Gehrmann et al. (2019); Mireshghallah et al. (2024) proposed detection methods that rely on language models token distribution. For the supervised detection, Verma et al. (2023) proposed an approach that relies on the availability of labelled datasets of synthetic and human-written texts.
Chakraborty et al. (2023a) propose a 6-way split of synthetic text detection methodologies: (i) watermarking, (ii) perplexity estimation, (iii) burstiness
3https://github.com/gpucce/synthetic_llm_data
estimation, (iv) negative log-likelihood curvature, (v) stylometric variation, and (vi) classifier-based approaches.
After the release of ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), a lot of studies (mostly not yet peer reviewed) focused specifically on the detection of text generated by these models (Dhaini et al., 2023). Since it is not possible to access token probabilities for a candidate generated text, some of this work relies on a proxy model (Vasilatos et al., 2023), but most rely on supervised classification (Mitrović et al., 2023; Liao et al., 2023; Guo et al., 2023; Liu et al., 2023), including OpenAI itself (Kirchner et al., 2023). Sometimes classical machine learning techniques are reported to perform well: e.g. Desaire et al. (2023) report a high performance on chemistry articles with XGBoost classifier and 20 features extracted from paragraphs.
Recently, several benchmarks for the synthetic text detection task have been released, Wang et al. (2024b,a) contribute a large multilingual, multidomain dataset that can be used to benchmark synthetic text detection systems. Macko et al. (2023) propose a benchmark for the detection of text generated by multilingual LLMs, while Dugan et al. (2024) propose a very large benchmark for synthetic text detection, controlling for the temperature used to generated the text, which has been shown to be relevant for detection (Mitchell et al., 2023).
The above works focus on the scenario where the effort of detecting synthetic texts is on the user side. The complementary direction on the developer side is watermarking: ensuring that the LLM output creates some kind of statistical “signature” that would help to identify it. There are multiple proposals for how to do this (Fernandez et al., 2023; Kirchner et al., 2023; Kuditipudi et al., 2024; Li et al., 2023; Takezawa et al., 2023; Wu et al., 2023; Yoo et al., 2023, inter alia), and some initial results suggesting that such techniques could be sufficiently robust to human and machine paraphrasing (Kirchenbauer et al., 2023, 2024). However, most of the current ‘open’ LLMs are freely available without any watermarking, and as we will show, they are already sufficient to be used as CFMs.
For the problem of detecting synthetic text without watermarking, the current research focuses on either monolingual or multilingual-by-design LLMs, and most studies do not focus on a specific domain. We stress the importance of also investigating fine-tuned models because LLMs are expensive to both train and run inference on (Samsi
et al., 2023). Hence, starting from a public, relatively small, but high-quality LLM and fine-tuning it for a specific type of content is probably the most plausible scenario for ‘content farms’ that aim to produce texts cheaply. To the best of our knowledge, this is the first study to focus on the scenario where the ‘CFM’ is fine-tuned for news, and in a language that it was not meant for originally. Moreover, except for the recent work by Wang et al. (2024a), existing resources do not cover Italian.

heading: 3 Fine-tuning of Llama as an Italian ‘Content Farm’ Model
text: As our ‘CFM’, we choose the original Llama model (Touvron et al., 2023a), in 7B and 65B parameter versions. Since this is one of the first public highperforming LLMs of this size, our Llama results serve as a lower bound for what could be expected from later LLMs, such as Llama 2 (Touvron et al., 2023b) and Llama 3,4 Aya (Aryabumi et al., 2024), and others. We do not suggest that it is possible to obtain good results with any language and any LLM (see also §10): such a transfer depends on the similarity between languages and their coverage in the training data. But given that ‘content farms’ have already been identified in at least 16 languages (see §1), many others could follow.
Our choice of Italian enables a lower-bound estimate of what could be expected of monolingual or multilingual LLMs with more exposure to the target language. Llama is a ‘mostly-English’ model, not intended to be multilingual. The resources it was trained on, such as C4 corpus (Raffel et al., 2020), made deliberate efforts to filter out non-English text. However, it was exposed to at least Italian Wikipedia (Touvron et al., 2023a, p.2). Hence, Wikipedia is likely the main, if not the only source of Italian in Llama.
We experiment with the original Llama baseline (65B pre-trained model with no extra training), and two versions of our Llama fine-tuned on Italian news, after 20K and 60K steps. The technical details for fine-tuning are provided in App. B.1. We remark that creating such a CFM now comes with very few technical or financial difficulties,5 which
4https://ai.meta.com/blog/meta-llama-3/ 5Renting GPUs on cloud providers such as Amazon is now relatively cheap (approx 15$ per hour for 8x40Gb A100) and would require as little as 100$ to replicate one of our LLM training sessions and data generation. The technical barrier to fine-tuning LLMs is also low now, thanks to tools like Huggingface’s Autotrain:https://huggingface.co/autotrain. We do not criticize open-sourcing such tools, but we hope that
could increase the number of bad actors.

heading: 4 Detection of Synthetic News in Italian
text: By Native Speakers
Methodology. To assess whether native Italian speakers would be able to identify synthetic news texts generated by our CFM, we set up a crowdbased study following the general recommendations for human evaluation of automatically generated texts proposed by van der Lee et al. (2021). Specifically, we created 4 surveys, with 100 questions in each. To maintain the rater engagement, we administered the surveys in five sessions, each comprising 20 questions. The raters were anonymously recruited among Italian native speakers via the Prolific6 online crowd-sourcing platform. Five different raters participated in each survey session, with no limitations on the number of sessions a rater could undertake. The 20-question sessions took 8 mins 23 secs on average, and the raters were compensated at 9,68$ per hour.7 The study involved a total of 93 different raters, with an average age of 32.01 (±10.76).
Each of the 4 surveys is designed to assess the texts generated by a different model: Llama 7B and 65B, both with and without fine-tuning on Italian news. Each question required raters to read two texts, denoted as A and B, and answer the question “Text B follows text A, do you think text B is written by a machine?”. The answer was a rating on a 5-point Likert scale, where 1 indicates ‘certainly human-written’ and 5 ‘certainly machinegenerated’. In 50% of the questions, both text A and B were human-written news articles coming from the original CHANGE-it test dataset.
The topics included daily national political events (e.g. politicians’ declarations), general news (e.g. climate catastrophes), and relevant international news (e.g. European leadership meetings).
To estimate the accuracy of human raters on this task, we map the scores on the 5-point scale to a binary score by computing the average score assigned by the raters to a given sample. We interpret the average score above 3 as indicating that a given sample B was generally perceived as machinegenerated. We also experimented with a different
our results would highlight the necessity of more research on synthetic text detection.
6https://www.prolific.co/. The participant group was balanced in terms of gender (49.46% female) and student status (50.60% reported being students).
7This hourly payment rate was certified as ‘Fair’ by the Prolific platform.
thresholding approach, using the average rating as a threshold, that showed similar results; these results are available in App. C.
Results. Table 1 shows the outcome of our analysis. Since the raters’ accuracy in detecting news generated by the largest fine-tuned Llama 65B is as low as 64%, we can answer our research question positively: Llama can be fine-tuned to generate hard-to-detect news-like text in Italian. We only used 40K samples for fine-tuning a relatively old model, so even more plausible-sounding synthetic text could likely be created with more data and more recent LLMs.
Overall, the raters’ accuracy reflects two foreseeable trends: the smaller 7B models are easier to detect, and the fine-tuned models are harder to detect. Interestingly, the small 7B version, when fine-tuned on Italian, is identified by raters with accuracy close to the larger pretrained 65B.
Table 1 also reports the average inter-rater agreement for each survey. Fleiss κ (Fleiss et al., 1971) is in the range between 22%-36%, indicating a “moderate” agreement (Landis and Koch, 1977) consistent with similar human-evaluation studies (van der Lee et al., 2021). The raters agree more strongly when assessing non-fine-tuned models, possibly because they occasionally switch from Italian to English mid-generation, a characteristic identified by raters as indicative of machine-generated texts. Figure 2 shows an example of such a switch.
The qualitative analysis of 100 machinegenerated instances (25 per model) showed that 46/100 examples had no obvious issues with language, but, as expected for LLM-generated texts, their content was factually incorrect and, worryingly, the factual errors were not necessarily obvious without extra fact-checking effort. Additionally, in this sample of 100 texts, we found 7 examples where the generated text contradicted the prompt, 8 cases of language-switching, 18 samples
with grammatical errors and 21 with expressions that are grammatically correct but unnatural in Italian. Annotated examples for each model are shown in App. D.
We stress that this study focuses only on the problem of synthetic news-like text, which is sufficiently plausible for the ‘content farms’ to lead the users to ‘news’ websites and be served ads. Their success also likely depends on the quality of the ‘headlines’, their match to the interests of the audience, the position in the search engine rankings and other factors beyond the scope of this study. Still, having such texts is a necessary, though not sufficient condition for operating a ‘content farm’.

heading: 5 Automatic Detection of Synthetic News in Italian

heading: 5.1 Approaches based on token likelihood
text: Methodology. Similarly to Jawahar et al. (2020); Sadasivan et al. (2023); Chakraborty et al. (2023b), we attempt the zero-shot detection of artificially generated text. We experiment with two approaches for synthetic text detection. Both assume to have access to the likelihood of each token in a sentence, according to the model whose “authorship” is under analysis.
The log-likelihood measures how likely a sentence is according to the probability assigned by the model to each token. The core idea behind the DetectGPT score (Mitchell et al., 2023) is getting a more robust score by normalising the log-likelihood of a sentence based on modifications8 of that same sentence generated by a different model (which we
8In our case, we are interested in the likelihood of a synthetic sentence estimated by Llama, vs the normalized likelihood that Llama assigns to modifications of that same sentence that we generate with T5 (Raffel et al., 2020) as the bootstrap model. See App. D for examples of such modifications.
refer to as the bootstrap model). Given a sentence, both log-likelihood and DetectGPT can be used with a threshold to tell if that sentence is more likely written by a human or by a language model. The threshold is estimated empirically based on a collection of synthetic and real samples. We refer the reader to Mitchell et al. (2023) for details of likelihood-based methods.
We start by validating our codebase and implementation by repeating prior experiments on xsum (Narayan et al., 2018) dataset in English, and we obtain comparable results (see App. E).
To measure the ROC of both DetectGPT and log-likelihood we select 100 random sentences and generate9 another 100 by using the first 30 tokens of each sentence as a prompt for the model under analysis. After this, we clip all sentences to 150 tokens,10 measure the score for each sample, and compute the AUROC on all 200 sentences (half human-written and half machine-completed).
To compare the detection approaches for the original and fine-tuned Llama, we use the CHANGE-it test dataset for Italian. As the bootstrap model, we use IT5-large (Sarti and Nissim, 2022) for Italian.
Results. Figure 3 shows the AUROC of DetectGPT and log-likelihood for all our models. loglikelihood does not seem to react to fine-tuning at all: the fine-tuned models have almost the same AUROC as the pre-trained one. DetectGPT does have a 5-6 points higher AUROC for the fine-tuned models. However, by qualitative analysis, see §4,
9All generation tasks are performed using Nucleus Sampling for decoding (Holtzman et al., 2020).
10Due to different tokenizers, this step results in sentences with a varying number of words, but a similar length.
we find that fine-tuning should have made the task more rather than less difficult. Without fine-tuning, our Llama CFM has a tendency to switch to English mid-generation, as we also observed this in the human detection study (see Figure 2, more examples available in App. D). This language switch should be a clear marker for detecting both the original 7B and the 65B models, and it vanishes after finetuning. But both log-likelihood and DetectGPT are missing this clear signal.
Although the DetectGPT and log-likelihood perform relatively well in our tests, we stress that this indicates a measure of the difficulty of this task, rather than a solution to synthetic news detection (see section 7). We remark that DetectGPT score can be turned into an accuracy measurement by fixing a threshold, for a direct comparison with the human evaluation accuracy. In our case, DetectGPT accuracy is ≈ 80% using the median11 score as the threshold for fine-tuned Llama 65B.

heading: 5.2 Supervised Detection of Synthetic Texts
text: A different approach to identifying synthetic texts is to train supervised classifiers (Liu et al., 2019; Conneau et al., 2020). However, this requires a labelled and balanced dataset of human and synthetic texts. To understand the challenges of this scenario, we use DICE, a different Italian news dataset focusing on crime news (Bonisoli et al., 2023). We mix DICE with the CHANGE-it data as well as the synthetic texts. This simulates a more realistic data collection process compared to previous studies that solely focused on fully in-domain data.
11We choose this threshold knowing that the dataset is balanced and that DetectGPT is monotonic, otherwise we would need to tune it.
Methodology. To create synthetic news, we finetune the following recent models: llama2-7b, llama2-13b (Touvron et al., 2023b) and Mistral-7b (Jiang et al., 2023) on the full CHANGE-it training set (see App. B.2 for the fine-tuning details). We refer to the resulting fine-tuned models as llama-27b_it, llama-2-13b_it, and mistral_it.
For each of these models, we create a suite of datasets with training sets composed of 2K, 4K or 8K samples. In all settings, the test sets comprise 2K samples, namely 1K texts from the CHANGE-it test set and 1K synthetic news with the same titles. The training sets are built in two ways. In in domain setting, 50% texts are synthetic, and 50% are articles sourced from CHANGE-it. In mixed source, the human-authored articles come from two different datasets, 25% each (CHANGE-it and DICE). Mixed source is closer to the scenario where we do not know what dataset was used to fine-tune the CFM, and sample from a wide range of possible news articles. The more diverse this set of nonsynthetic examples, the harder the classification task will probably become.
Similarly to Wang et al. (2024b), we experiment with two classifiers, based on RoBERTa (Liu et al., 2019) and XLM-RoBERTa (Conneau et al., 2020). The former is pre-trained on English and the latter
specifically made for multilingual settings. We use the same hyper-parameters for both classifiers, and we train them for 3 epochs. The technical details are provided in App. B.3.
Results. Figure 4 shows that at 8k training samples xlm-roberta-large accuracy ranges between 84% and 92% depending on the generator. With 4k training samples, the accuracy decreases between 81% and 86%, with a similar difference between in domain and mixed source. With 2k training samples, the performance for all but 3 models drops to under 56% – i.e. almost random chance level. The RoBERTa-based classifier yields a similar trend. See App. B.3 for the graph for RoBERTa and numerical results for both experiments.
We find that for these datasets there is a threshold between 2k and 4k samples that marks a strong difference in the performance of the supervised classifier (20-25% loss in accuracy), while increasing the dataset size further provides diminishing returns (2-8% gain). Our mixed source scenario at 8K samples is also consistently harder for the classifier; at 4K the results are mixed across different LLMs, and at 2K it is hard to draw firm conclusion for llama-2-7b_it and mistral_it because their performance is generally low. This is with a multilingual classifier base; for RoBERTa-based classifier, the mixed setting is consistently harder in all settings (see App. B.3).

heading: 6 Detecting CFMs with Proxy Models
text: 6.1 How Much Fine-tuning Do We Need? So far we only considered the scenarios where the model generating synthetic texts is the same that computes the likelihood used to detect them. Let us now consider an alternative: a proxy model approximating the likelihood scores of the CFM. This could be expected to work if the proxy model and the CFM were fine-tuned on the same dataset. But that assumption is also too strong to be practical.
We explore relaxing it further to the scenario where we only have access to a small set of texts similar to the fine-tuned model outputs. This could be a small subset of the fine-tuning dataset or, more generally, samples from a similar distribution (e.g. different samples from a given newspaper or a social media account). We ask whether this is enough: can we train a model on a small set of in-distribution texts, so that the likelihood it assigns to tokens is sufficient to detect texts from a fully fine-tuned CFM?
Detector model Generator model
llama-2-13b_it llama-2-7b_it mistral_it dGPT llh dGPT llh dGPT llh
Methodology. To answer this question, we finetune the three LLMs used in the supervised detection experiment in §5.2 (namely llama-2-7b_it, llama-2-13b_it and mistral_it) on varying subsets of the CHANGE-it dataset. We then use their likelihood and DetectGPT score to detect synthetic text from models fine-tuned on the full CHANGE-it dataset, as described in §5.
For that, we select 5000 samples from the CHANGE-it test set, and for each sample, we prompt all models with the title and initial tokens of the original article (see Appendix F for more details). This results in 3 datasets with 10k samples in each, 5k human written (from the CHANGE-it test set) and 5k synthetically generated news with the same titles, which can be used as a benchmark for the detection of synthetic news in Italian.
Afterwards, we select two subsets of the CHANGE-it training dataset with 3981 and 7862 samples, respectively 3% and 6% times the original fine-tuning dataset size. We fine-tune the same LLMs on both of these smaller training sets, see App. B.2 for details on the fine-tuning procedure and Table 4 for a summary of the model naming.
Finally, for each of the three synthetic datasets generated with llama-2-7b_it, llama-2-13b_it and mistral_it, we compute log-likelihood and DetectGPT score with all 12 models: 3 fine-tuned on the full CHANGE-it dataset, 3 fine-tuned on 3981 samples, 3 on 7862 samples, and the original LLMs without any fine-tuning.
Results. Table 2 suggests that the answer to our research question is positive: a small subset of fine-tuning samples is indeed sufficient to detect a full fine-tuned model. A model fine-tuned on only 3% of the fine-tuning dataset achieves between 86.1% and 95.4% of the AUROC measured for the fine-tuned one.
However, this is only effective if the same LLM is both the generator and the detector, see App. G for the ROC curves in this case. Simply finetuning different LLMs does not make them similar enough to use one for detecting another. In the case of mistral_it, it actually gets worse after fine-tuning (we hypothesize that it could be due to differences in tokenization).
While this study focuses on the Italian news, in App. E.2 we perform the same experiments on the XSUM dataset in English, and come to equivalent conclusions: we can detect a fully fine-tuned model with a model that is fine-tuned on a small subset of the whole fine-tuning dataset.

heading: 6.2 Can Ensembling Help?
text: As shown in Table 2, fine-tuning on few samples is sufficient to achieve strong AUROC both with loglikelihood and DetectGPT on fine-tuned versions of the same LLM – but detecting fine-tuned versions of different LLMs is harder, and longer fine-tuning does not improve the performance of statistical detection methods. This makes the proxy model approach impractical, since in the real world we would not know which base LLM was used.
To address this limitation, a straightforward ap-
proach would be ensembling the DetectGPT score, which different candidate LLMs assign to a given sample. If texts from a certain source consistently get a high AUROC, this would signal that they are probably synthetic. This is reasonable on the assumption that the CFM is one of the recent highperforming open-source LLMs, and there are not too many of these. Still, to evaluate the potential of this approach we experiment with computing both the mean and max DetectGPT score among three LLMs with the same fine-tuning, to understand if this provides a score with higher AUROC.
The results are shown in Table 3. Neither the average nor the maximum DetectGPT score offer a simple solution, but in most cases, one of them yields a significantly higher AUROC than the random guess. We believe that this is overall a promising direction for developing new statistic approaches based on mixing the likelihoods of several models, but, once again, it would be difficult to scale to dozens of candidate LLMs.

heading: 7 Discussion: Are There Practical Solutions to Synthetic Text Detection?
text: As described in §1, the ‘content farms’ for news are already wide-spread, and it is in the public interest to identify such ‘outlets’, as soon as possible.
One key problem is that we would not know which base LLMs were used, and we would not have access to their token likelihood information. This completely precludes using methods like DetectGPT (§5.1). Our proxy model approach (§6) could address the latter problem, but the former
is far from being solved (§6.2), and will only get harder as more LLMs are published. The offending CFM could also be a ‘closed’ model like GPT-4, to which we would never have white-box access.
Supervised approaches can be used, but, as we showed in §5.2, they rely on a relatively large12 dataset of texts that were manually identified as CFM texts, and human-written texts to serve as ‘negative‘ examples, and it is difficult13 to get the human-written samples right. It would take time, expertise, and resources to develop such a dataset. And we might want to detect a CFM before it publishes even 2K “news”. The synthetic examples could also come from multiple CFMs, making the classification task even harder. It is telling that the classifier developed by OpenAI, presumably aiming to detect only OpenAI models, was soon shut down due to low accuracy (Kirchner et al., 2023).
Concerning watermarking, in the CFM case, it is safe to assume that the deployers of such models would likely try to remove or obfuscate the watermarking, which is relatively easy to do by altering the generation strategy. And the public easy-to-use tools highlighting the statistical ‘evidence’ of the watermark14 could be used not only by those looking to detect CFMs, but also the CFM operators, to obscure the evidence.
We conclude that at the moment there are no practical options for detecting news ‘content farms’ in the wild, and both the open-source LLM community and providers of ‘closed’ LLMs need to consider ways to address that. We are hopeful that some combination of centralized watermarking effort and further development of detection methods could provide a working solution in the future.
Since the hardest problem is identifying the base LLM, one policy direction to consider would be to (a) mandate watermarking, ideally built-intomodel-weights, as a pre-requisite for either commercial deployment of LLMs or their open-source publication, (b) maintain a public watermark detection library, allowing to identify a candidate LLM given a text sample. This would not be bullet-proof,
12Verma et al. (2023) used 30k samples for English alone. 13As a binary classification task, the detection of synthetic texts from a particular CFM seems intrinsically very challenging, since ideally we would need the non-synthetic examples to accurately represent all possible non-synthetic sources, and also to be matched with a comparable number of synthetic examples.
14E.g. demo by Kirchenbauer et al. (2023) at https://huggingface.co/spaces/tomg-group-umd/ lm-watermarking
but it would raise both the costs of avoiding detection and the awareness of the problem in the NLP community.

heading: 8 Conclusion
text: In this work we have shown how creating a ‘content farm’ generative model for news-like text can be easy, even though we started with a relatively old LLM and a language it was not originally meant for. After fine-tuning Llama 65B on only 40K Italian news texts, native speakers of Italian have only ≈ 64% accuracy on the synthetic news text detection task.
We show that the current approaches to automatic text detection, based on token likelihood and supervised classification, outperform human raters in the synthetic text detection, but they would all be impractical in the real world, since they require access to the token likelihood information or a large training dataset. We further consider the proxy model approach, and we find that it works well even with little data for fine-tuning, but only if it is known which base LLM was used. Our study highlights the urgency of further work on developing model-agnostic methods of synthetic text detection.

heading: 9 Acknowledgements
text: We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support IsCb2_GELATINO (HP10CQRW2J) and IsCb3_TRAVEL (HP10CY9V7K).
This work was also partially supported by FAIR (PE00000013) project under the NextGenerationEU programme, partially by the PNRR project ITSERR (CUP B53C22001770006) and partially by the Project PRIN 2022EPTPJ9 (WEMB – “Word EMBeddings: From Cognitive Linguistics to Language Engineering, and Back”), funded by the Italian Ministry of University and Research (MUR).
The authors’ opinions do not necessarily reflect those of the funding bodies.

heading: 10 Limitations
text: Generalizability to other languages. We present a case study on a single language, and do not intend to claim that it is possible to generate plausible-sounding text in any language, by fine-tuning a mostly-English model like Llama. But our results suggest that it may be possible, at
least for languages with a similar level of coverage in datasets used for training LLMs. More research is needed to establish both the factors impacting the success of such transfer, and better methods to detect synthetic texts.
For the original Llama, according to Touvron et al. (2023a, p.2), it was exposed to Italian Wikipedia in pre-training. Italian Wikipedia currently has about 500K articles.15 For other sources included in Llama, such as C4 (Raffel et al., 2020), we cannot exclude the possibility that there was some Italian – but deliberate effort was made to filter out non-English texts, and so we assume that there was at least not much contamination. Other languages in Llama with about the same amount of Wikipedia data as Italian are Polish and Dutch (≈ 500K articles). In the ≈ 400K range there are Spanish and Portuguese, and at about ≈ 300K – Russian and Swedish. A future study could explore how the amount of Wikipedia data, the amount of fine-tuning data, and the typological distance from English impact the success of the transfer.
It is of course also possible and likely that a CFM developer aiming for a specific language would start with an LLM that is multilingual by design, such as BLOOM (Scao et al., 2022) or AYA (Aryabumi et al., 2024), and probably get even better transfer.
Other methods of detecting synthetic text. In the scope of this paper, we experimented with two statistical detection methods (based on likelihood scores and supervised detection), but there are others, including more sophisticated likelihood based approaches (Su et al., 2023) that however share the same dependency on the likelihood of the original models, and therefore the core limitations of the methods we experiment with. This does not invalidate our conclusions and the general answer to our research question, but it could be expanded in the future work.
Limitations of the human evaluation. Our selection of human raters was based solely on Italian as their native language. Future work could investigate whether the results would differ across different occupations and education levels, and with different kinds of synthetic and real articles.
Our human evaluation protocol considers the setting where the model is prompted with the first
15https://en.wikipedia.org/wiki/Wikipedia: Multilingual_statistics
30 tokens of a real human-written article, because the model is not trained using the articles headlines but just to generate news, to make the adaptation from English to Italian simpler. Another scenario to be tested in future work is when the model is prompted with headlines (authored by the content farm owner or auto-generated). That could affect the quality of the generated text or the ease of its detection.
Finally, our study focuses on the possibility to create plausible-sounding news-like text that could be used by “content farms”, rather than text created for specific misinformation campaigns or to spread conspiracy theories. It is possible that, similarly to human-authored fake news, the human raters would be more likely to doubt the authenticity of the article when it had some big factual claims that were easy to check. This factor also remains to be explored in future work.
The best case scenario with a CFM is that the users would soon realize that the site is fake, and leave – but even in that case they would already have wasted time and resources, and potentially increased their digital footprint because of tracking on the CFM website. Further possible harms from misinformation, manipulative targeted ads etc. are beyond the scope of this study, and will require a more detailed investigation of various types of factually problematic content, deliberate attempts to introduce certain narratives, awareness and training of the users, etc.
Detection of API-based models. This study focused on the scenario where the ‘content farm’ used its own model, created by fine-tuning a publicly available high-performing base LLM. It could of course also use an external API service, which would make its task even easier technically.
Famously, GPT-2 (Radford et al., 2019) initially came with warnings about it being “too dangerous to release”, precisely because of the danger of synthetic ‘news’ (Wakefield, 2019). Section 7.4 in the GPT-3 (Brown et al., 2020) report is dedicated to synthetic news generation and the finding that humans detect such 200-word GPT-3 texts in the 52%-76% accuracy range. At that time, too, there was coverage of the dangers of synthetic news (e.g. Mak, 2019; Knight, 2021), but no policies ensued.
Now that OpenAI offers the most popular generative AI services, its proposed solution is its Terms of Service, presumably enforced via constant monitoring of the API use by all users. Its current Terms
of Service broadly prohibit “any harmful, illegal, or abusive activity”, but it is not immediately clear which definitions for ‘harm’ and ‘legality’ must be followed, and whether the news ‘content farms’ websites are covered. The most directly relevant clause currently16 seems to be representing “that Output was human-generated when it was not”. We do not know how this is enforced, and how many bad actors are successfully stopped with API-level controls – but clearly not all of them.17

heading: 11 Broader Impacts
text: Impact on society. This work aims to highlight a potential problem for the information infrastructure of worldwide communities, that may currently consider themselves safe from plausible-looking synthetic text due to the lack of high-quality monolingual models for their languages. We show that a relatively old Llama model, exposed only to Italian Wikipedia and 40K news articles for fine-tuning, is sufficient for generating very plausible-looking synthetic news in Italian, and there are no practical solutions for detecting such text. We hope that this work would spark similar investigations for other languages, and highlight the urgency of development of reliable and model-agnostic methods for detecting synthetic text.
In particular, we hope to draw the attention to the fact that at present, the most authoritative source of information about the extent of the problem with the ‘content farm’ news websites seems to be the aforementioned NewsGuard reports (Sadeghi and Arvanitis, 2023; Sadeghi et al., 2024). They are based on extensive expert research and manually vetting different news outlets, and are provided as a paid service. Ideally, the society would be better informed about the scope of the problem18, have a reliable public infrastructure for news resources that come from real outlets with editorial responsibility, and taking quick action on the misleading
16https://openai.com/policies/terms-of-use, accessed on Feb 10 2023.
17For example, Gizmodo identified a fake story about the death of Joe Biden that started with “I’m sorry, I cannot complete this prompt as it goes against OpenAI’s use case policy on generating misleading content. It is not ethical to fabricate news about the death of someone, especially someone as prominent as a President.” (DeGeurin, 2023)
18The fact that number of such websites grew so quickly in the past year (from 49 to 840, see section 1) must mean that they are sufficiently profitable. Hence, a large number of people must be mislead at least to waste their time and resources on visiting a spammy website, and there are other possible harms (e.g. resulting from misinformation).
AI-generated websites. The problem with the current wave of such ‘news’ originates in the field of NLP, and we hope that our field can also contribute practical solutions to the problems of detecting synthetic texts and assisting with their reporting.
Human and computational resources. This work is based on the publicly available models (Radford et al., 2019; Touvron et al., 2023a,b; Raffel et al., 2020; Sarti and Nissim, 2022; Jiang et al., 2023) and resources (Narayan et al., 2018; Mattei et al., 2020), and documents its emissions (App. H), annotation procedure and compensation to the human raters (§4). The code to reproduce our experiments accompanies the submission will be publicly available with the publication of the paper.

heading: A Model Sources and Naming Scheme
text: All LLMs used in this study are listed in Table 4. We also add name of the models in HuggingfaceHub, e.g. for meta-llama/Llama-2-7b the pretrained model comes from https://huggingface. co/meta-llama/Llama-2-7b.

heading: B Fine-tuning Details
text: B.1 Llama 1
We fine-tune both Llama 7b and 65b models on a randomly chosen 40K subset of the CHANGE-it news dataset. The articles are arranged in training sequences composed of 128 tokens, adding subsequent segments one after the other.
The training was performed on 8 nodes, each with 4 V100 GPUs with 16GB VRAM. The effective batch size is 128, real batch size 2, 64 accumulation steps. The maximum learning rate is 0.0005, with a one-cycle scheduler without warmup.
Due to memory constraints, instead of the most effective AdamW optimizer we use simple Stochastic Gradient Descent. We train for 60,000 steps (120,000 samples split into 3 epochs of 40,000) saving checkpoints every 10,000 steps (20,000 samples). This takes approximately 3 days.
B.2 Llama 2 and Mistral
We fine-tune Llama 2 7b and 13b as well as Mistral models on the full training set of the CHANGE-it news dataset. With a length up to 2048 tokens.
The training is performed on 4 nodes, each with 4 A100 GPUs with 64GB VRAM. The effective batch size is 256, real batch size 4 with 4 accumulation steps. The maximum learning rate is 1.e-5, with a one-cycle scheduler without warm-up.
We use AdamW optimizer with FSDP, we train for three epochs in bfloat16 to limit the memory
needed and we perform full fine-tuning without using any parameter efficient technique.
For the fine-tuning on subsets of the CHANGEit dataset we keep most things equal to the longer training-set.
For the XSUM dataset the same setting is kept almost identical.
B.3 Fine Tuning Supervised Synthetic Text Detectors
We fine-tune two classifiers to identify real or synthetic texts, RoBERTa-large and XLM-RoBERTaLarge. The models are trained on three different dataset size, 2k, 4k, 8k and two data mixing.
The max learning rate is 5.e-5 we use the a batch
size of 128 and a linear decaying learning rate without warmup. The rest of the parameters are the daults parameters used by Huggingface Trainer.
The performance is reported in Figure 4 for XLM-RoBERTa and Figure 5 for RoBERTa.
The detailed numerical results are also listed in Table 5.

heading: C Alternative Threshold for Human Evaluation Metric
text: To compute the readers accuracy on identifying machine generated texts, we threshold the average score assigned to a sample to obtain a binary label. In §4 we show the results using 3 as a threshold, which is the mean possible rating, but we find that the results are similar when the threshold is the scaled mean: the mean score of all questions in a 100-question survey. Table 6 shows all results with this threshold, and they are well aligned with Table 1.

heading: D Generation examples
text: This appendix lists further examples of LLMs pretrained mostly on English switching from English to Italian when generating the text, as well as the same examples where our CFMs don’t do so. We show this for Llama 7b not-pretrained Table 8, Llama 7b fine-tuned Table 9, Llama 65b pre-trained Table 10 and Llama 65b pre-trained Table 11. The captions also report a break down of the output of the qualitative study we carried out, listing the amount and type of mistakes we spot.
Furthermore, Table 12 shows examples of the changes done by it5 when generating the modifications necessary to compute the DetectGPT score. These modifications can sometimes lchange meaning in Italian, although when the modified spans are only 1 or 2 words long they keep their original meaning.
D.1 Modifications from IT5 The modification from IT5 to compute the DetectGPT score don’t necessarily need to have a meaning as they are only used to normalize the loglikelihood values, however, it is interesting to note that often they don’t disrupt the sentence meaning, and that they can create new sentences, although with similar meaning, that can themselves be useful as synthetic data. We release 600k of these sentences that are half modifications of the original news and half modifications of the synthetic texts generated by our fine-tuned models.

heading: E Generalizability to English
text: E.1 Synthetic Text Detection Based on Token Likelihoods
To confirm that our setup in subsection 5.1 is comparable to the previous efforts, we start by replicating a core result by Mitchell et al. (2023): on xsum data (Narayan et al., 2018), and using different versions of t5 (Raffel et al., 2020) DetectGPT outperforms the log-likelihood in detecting GPT2 text (Radford et al., 2019) (see Figure 6a).
We apply the same methodology to sentences obtained using Llama 13B and Llama 65B. For Llama, we were unable to get DetectGPT to achieve a higher AUROC than the log-likelihood. We believe this to be due to the stronger performance of Llama compared to the t5 model used to generate new sentences (Figure 6b). This suggests that English text generated by Llama is harder to detect.
To test this hypothesis we measure the importance of the bootstrap model in this case. Table 13 shows the AUROC of DetectGPT depending on the bootstrap model, a larger t5 model leads to higher AUROC.
We repeat the experiment with both DetectGPT and log-likelihood at various temperature settings (0.6, 0.8, 1.0), and we find a strong sensitivity to this hyper-parameter, which merits further investigation.
To establish a fair comparison with DetectGPT while testing Llama, we perform an ablation study based on varying the temperature used in generation.
Table 14 shows different AUROC values for different temperatures. It appears that there is a strong sensitivity of the detection methodologies to this hyper-parameter, which merits further investigation in future work. The value 0.8 where DetectGPT and log-likelihood are more aligned, is also the value reported in the Llama repository.
E.2 Detecting CFMs with Proxy Models
We repeat the experiments done for the CHANGEit dataset also for the XSUM dataset (Narayan et al., 2018).
That is we fine-tuned llama-2-7b and mistral on the full XSUM training set. We generate 2 datasets with 1k synthetic texts generated with each of the fine-tuned models and 1k samples from the xsum test set (less than for CHANGE-it to limit compute costs). Then we also fine-tune these two models on a small subset of the trainig set, 995 samples and finally we compute DetectGPT achieved by these 6 models, llama-2-7b and mistral pre-trained, llama-2-7b_xsum_995 and mistral_xsum_995 finetuned on 995 samples and llama-2-7b_xsum and
mistral_xsum fine-tuned on the xsum dataset. Table 15 shows the results that closely match those for CHANGE-it shown in Table 2, namely that fine-tuning also on a small set of the same domain leads to high AUROC while if models come from different pre-trained ones the performance is low.

heading: F Prompting details
text: Figure 7 shows the exact prompt that was used for generating the synthetic texts in section 6. In the prompt, we retain a few initial tokens of the original article, ensuring that the prompt never exceeded 30 words in total.

heading: G Receiver Operating Characteristic (ROC) Curves
text: The receiver operating curve is a measure to understand how setting a threshold over a score would influence the true positive rate (TPR, the numer of instances marked as positive divided by the total number of positives) and the false positive rate (FPR, the number of false positive divided by the number of all negatives). That means how setting all instances with a score above a certain threshold as positive would influence the number of true positive and false positive.
Given that when we set as a threshold the maximum value of a score all instances are classified as negative TPR equals 0 and FPR equals 0, viceversa when it is set to the minimum TPR equals 1 and FPR equals 1. The ROC curve is a plot of the TPR against the FPR for different thresholds.
To show further details about the proxy models AUROC values shown in Table 2. Figure 8 shows the ROC curves for the case when the proxy models come from the same pre-trained one.
We can see tha computing the AUROC over 10k examples leads to a smooth curve and that the shape is very consistent across different amounts of finetuning futher validating the strngth of proxy models.

heading: H Computational Costs
text: The fine-tuning run of our ‘CFM’ Llama (§3) lasted 5 days, as we wasted approximately 2 days due to exploding loss. Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A cumulative of 2,304 hours of computation was performed, and total emissions are estimated to be 298.6 kgCO2eq.
The fine-tuning of llama-2-7b_it, llama-2-13b_it and mistral_it on CHANGE-it took 64 GPU hours each on A100 64Gb GPUs. With the costly synthetic data generation, all together resulted in approximately 2000 GPU hours. Experiments were conducted using the LEONARDO cluster, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A cumulative of 2000 hours of computation was performed on hardware similar to A100 PCIe 40/80GB (TDP of 250W). Total emissions are estimated to be 216 kgCO2eq (Luccioni et al., 2019).
Thus, we estimate that the total emissions for experiments in this study amount to approximately 515 kgCO2eq.

heading: Future Work
text: It appears that there is a strong sensitivity of the detection methodologies to this hyper-parameter, which merits further investigation in future work. The value 0. 8 where DetectGPT and log-likelihood are more aligned, is also the value reported in the Llama repository. E. 2 Detecting CFMs with Proxy Models
We repeat the experiments done for the CHANGEit dataset also for the XSUM dataset (Narayan et al. , 2018). That is we fine-tuned llama-2-7b and mistral on the full XSUM training set. We generate 2 datasets with 1k synthetic texts generated with each of the fine-tuned models and 1k samples from the xsum test set (less than for CHANGE-it to limit compute costs). Then we also fine-tune these two models on a small subset of the trainig set, 995 samples and finally we compute DetectGPT achieved by these 6 models, llama-2-7b and mistral pre-trained, llama-2-7b_xsum_995 and mistral_xsum_995 finetuned on 995 samples and llama-2-7b_xsum and
mistral_xsum fine-tuned on the xsum dataset. Table 15 shows the results that closely match those for CHANGE-it shown in Table 2, namely that fine-tuning also on a small set of the same domain leads to high AUROC while if models come from different pre-trained ones the performance is low