LLM Detectors Still Fall Short of Real World: Case of LLM-Generated Short News-Like Posts

Authors: Andrei Kucharavy, Henrique Da Silva Gameiro, Ljiljana Dolamic
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Detection, misinformation, benchmarking

Abstract:
With the emergence of widely available powerful LLMs, disinformation generated by large Language Models (LLMs) has become a major concern. Historically, LLM detectors have been touted as a solution, but their effectiveness in the real world is still to be proven. In this paper, we focus on an important setting in information operations—short news-like posts generated by moderately sophisticated attackers.

We demonstrate that existing LLM detectors, whether zero-shot or purpose-trained, are not ready for real-world use in that setting. All tested zero-shot detectors perform inconsistently with prior benchmarks and are highly vulnerable to sampling temperature increase, a trivial attack absent from recent benchmarks. A purpose-trained detector generalizing across LLMs and unseen attacks can be developed, but it fails to generalize to new human-written texts. 

We argue that the former indicates domain-specific benchmarking is needed, while the latter suggests a trade-off between the adversarial evasion resilience and overfitting to the reference human text, with both needing evaluation in benchmarks and currently absent. We believe this suggests a re-consideration of current LLM detector benchmarking approaches and provides a dynamically extensible benchmark to allow it (https://anonymous.4open.science/r/text_llm_detector-3E07).

Decision: reject
Reviewer scores: 5, 3, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730052388196

thread_type:
official_review

<thread_object>
creator: reviewer_lmfw

<content>
summary:
This paper investigates the effectiveness of various LLM text detectors on short news-like posts on social media. It is shown that existing zero-shot detectors are easily fooled by trivial attacks, such as increasing the temperature or repetition penalty during the LLM text generation process. Detectors which are trained on specific LLMs may generalize to new LLMs but fail on off-distribution human-written texts. This suggests that current LLM detector benchmarks are insufficient and highlights the need for dynamic benchmarks that can be adapted to specific domains and threat models.

strengths:
1. Evaluations on off-distribution human texts which was missing in prior literature
2. It is interesting to see that some zero-shot detectors do not generalize to temperature increases/repetition penalty

weaknesses:
1. Limited novelty: There are no new insights or technical contributions offered in the paper, and many of the results are not surprising. It is well-known that zero-shot detectors can be very sensitive to distribution shifts (which is cited by the authors) and purpose-trained detectors may be overfit to their training distribution.

2. The authors do not measure or control for quality degradation and/or content preservation after the attacks. It is hard to make sense of the results as the attacker has a lot of power without these constraints. For example, higher temperature may make the model less fluent/coherent which may make it an impractical attack

questions:
I think it would be very interesting to explore the trade-off between generalization to off-distribution LLMs vs off-distribution human texts in more detail. An interesting direction here would be to identify subpopulations in human text which are prone to be classified as LLM generated, or vice versa. This would shed more light on when these detectors may fail and how they may be biased towards or against certain human groups or LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730187245623

thread_type:
official_review

<thread_object>
creator: reviewer_q7dz

<content>
summary:
This paper studies LLM generated content detection via (LLM-based) detectors. Zero-shot detectors are found to be vulnerable to simple evasion attacks, while a manually trained detector can overcome this issue but failed to not flag human-written texts as LLM generated.

strengths:
- The studied question is highly practical and is crucial for societal concerns. Exploring the limitation of LLM generated content detector can help flag early signals on the boundary of such event and help guide better design choices for future content detectors.
- The found effectiveness using simple evasion attack on zero-shot detectors is important, suggesting the need of improvement when constructing benchmarks for generated content detection.

weaknesses:
The weaknesses I spot are as follows.
- The coverage of experiments is a bit narrow. All experiments studied in this paper are totally based on the CNN Dailymail news dataset and its variants generated by LLMs. This dataset bias could lead to biased results as well. The study should incorporate a broader spectrum of candidate dataset types.
- The claim that trained detectors fail to generalize to unseen human-written texts needs further evidence. As illustrated in the paper, the Electra_RR model benefits from a dataset mixture training for improving TPR. This trend is not discussed for FPR, thus the failure mode on US vs British English detection may be a fact of lacking such data in the training procedure and the 5% FPR location step, and not because trained detectors are not capable of doing so.
- The writing could be improved. There are grammar issues and incomplete sentences throughout the paper. Besides, the misuse of \citet and \citep makes the paper hard to read.

questions:
- Will the dataset mixture training statement hold true for FPR? Can the failure in Table 2 be solved by incorporating both US and British style human-written texts during training, or by selecting a proper threshold using both styles of texts?
- Will the claims hold on datasets other than news? For example, on books or review similar to the categorization in [1]?

[1] RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537844202

thread_type:
official_review

<thread_object>
creator: reviewer_9dar

<content>
summary:
The paper presents an evaluation on the performance of existing LLM detectors in fake news detection. Both purpose-trained detectors and zero-shot detectors are benchmarked with the newly proposed datasets. They are tested on normal datasets, cross-domain generalization and adversarial evasion techniques.

strengths:
1. Fake news detection is a crucial issue in society.
2. A comprehensive review on the literature is provided.
3. New datasets for fake news detection are generated to evaluate existing LLM detectors.

weaknesses:
1. The presentation is ambiguous and flawed. The first section of introduction does not clearly explain the motivation, problem and method of this paper. Besides, there are some grammatical mistakes in the main text, making it harder to comprehend.
2. This work offers some observations in a real-world scenario of fake news detection, but it lacks insights or findings that can further promote the development of LLM detectors.
3. The experiments only take several small LLMs to generate fake news, which is an assumption of "moderately sophisticated attacker". However, the attacker can also adopt proprietary APIs, e.g., ChatGPT, to generate better information. The detection performance on text generated by more powerful models should be included.

questions:
My major concern is about the position of this paper. It is more like an evaluation of the application in a real world scenario of fake news detection. There is no fundamental problem or challenge in the field presented and addressed in the work. I am not sure if it's suitable to accept it to a conference like ICLR.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562421014

thread_type:
official_review

<thread_object>
creator: reviewer_52s7

<content>
summary:
This work evaluates some of the existing LLM-generated short-text detectors, including the zero-short and purpose-trained detectors, on a newly proposed short news-like generated post dataset (with attacks). This work conducted some experiments that revealed that the THREE zero-short detectors investigated show heavily degraded performance when encountering attacks, particularly for a simple attack by increasing LLM sampling temperature. Given these findings, the authors conclude current LLM detectors still fall short of real-world usage. 

In summary, this work contributes to a new short news-like dataset, involving three types of evasion attacks, and evaluating some of the existing detectors on this dataset. Finally, this work draws some conclusions based on their empirical findings.

strengths:
- A news-like dataset generated by six LLMs was developed in this work. The source models include three non-chat models, i.e., Phi-2, Gemma-2B, Mistral-0.1; and three chat models, i.e., Gemma-chat, Zephyr, and LLama-3-8B-Instruct. Regarding the generated text, the authors proposed to cut and take only the first 500 characters to reduce detection effects in terms of text length. 

- Three different types of attacks, i.e., changing generation parameters, conducting prompting attacks, and paraphrasing attacks were considered to evaluate some of the existing methods. Experiments show detectors evaluated show much degraded detection performance.

weaknesses:
The major weakness is that this work lacks substantial contribution, both in terms of technical contribution and surprising experimental observations or findings. Please see my detailed comments below.

- While a news-like dataset was proposed, the scope of this dataset is rather limited. Since a real-world (as noted in the paper title) application should not be confined to news only -- they can be more broadly e.g. story writing, academic writing, etc. Thus, the findings (and corresponding claims) may probably not hold anymore in a broader scope. A good benchmark should include as diverse subjects as possible (as we evaluate LLMs). I would suggest the authors refer to detection-related works for commonly used benchmarks, e.g. Fast DetectGPT and [1-5].

- Besides, the text length has been restricted to 500 characters, a rather short piece of text. Though this setting was intended to alleviate the impacts of text length in fake text detection, the real-world application, however, a large of amount of generated text can be rather long texts. As a result, the conclusion drawn from this dataset may be applicable to long-text detection scenarios. Indeed, short-short detection is much harder than their long text counterpart. Unfortunately, some methods specifically designed for short-text detection are not discussed or evaluated. As a good benchmark dataset, it is strongly encouraged to be thorough, and objectively reflecting different detection scenarios.

- The source models generating fake texts are far from enough, as you are considering the real-world usage, more advanced models, such as GPT3.5 Turbo, GPT 4, GPT 4o, Claude etc, are strongly supposed to be considered (because these are better performing models tend to be preferred in real-world applications). Otherwise, the benchmark may not be representative enough. Please refer to more recent literature (e.g. [1-5], and follow generally-adopted routines to include such advanced models.

- This work lacks a thorough literature review, the methods evaluated are far from enough to draw the conclusion that "LLM detectors still fall short of the real world", because many recent state-of-the-art detection methods have been missing! To make this claim and findings more solid, I suggest additionally considering [1-5] as your baseline methods. 

- In fact, all machine learning models can be vulnerable to attacks, this may not prevent their usage in the real-world scenario. For example, the existence of adversarial examples (in face recognition, object detection etc), a typical type of evasion attack, does not prevent the wide application of these models. In other words, I don't find the robustness issues (including some attacks mentioned in this work) a surprising finding, given that many of these existing works also reported their drawbacks to these attacks! This is a widely-known problem, rather than first reported in this work! As a result, this can limit the novelty of this work.

[1] Yang, Xianjun, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. "Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text." ICLR 2024.

[2] Mao, Chengzhi, Carl Vondrick, Hao Wang, and Junfeng Yang. "Raidar: geneRative AI Detection viA Rewriting." ICLR 2024.

[3] Tian, Y., Chen, H., Wang, X., Bai, Z., Zhang, Q., Li, R., Xu, C. and Wang, Y., 2023. Multiscale positive-unlabeled detection of ai-generated texts. ICLR 2024.

[4] Hans, Abhimanyu, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Spotting llms with binoculars: Zero-shot detection of machine-generated text." ICML 2024.

[5] Su, J., Zhuo, T.Y., Wang, D. and Nakov, P., 2023. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. arXiv preprint arXiv:2306.05540.

questions:
For my detailed comments, please see the weakness part. I suggest the authors clearly state the major contributions and findings in the rebuttal, e.g. what are major findings that were not mentioned in previous works, what are major differences of the proposed dataset, compared with benchmark datasets used by many existing fake tex detection works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683921511

thread_type:
official_review

<thread_object>
creator: reviewer_yju5

<content>
summary:
This paper examines the effectiveness of existing large language model (LLM) detectors in identifying short news-like posts generated by LLMs in real-world scenarios. The authors find that current zero-shot and purpose-trained detectors are inadequate, particularly when subjected to simple attacks like increased sampling temperature. They highlight the need for domain-specific benchmarking and propose a dynamically extensible benchmark to improve detector evaluation. The study concludes that LLM detectors are not yet ready for real-world application to counter LLM-generated disinformation.

strengths:
1. This paper studies the current LLM-generated text detectors can not work well and conduct benchmark experiments.
2. The attack testing method utilizes high temperature and repetition penalty to reduce the diversity of the produced text, making it more difficult for detection methods that rely on the lack of diversity of AI-generated texts.

weaknesses:
1. The novelty is limited and the results are not surprising.

questions:
1. How does varying the size of LLMs (e.g., 2B-7B parameters) impact the effectiveness of both detection and adversarial evasion?

2. One finding was that detectors trained on specific models generalized better across unseen LLMs but struggled with human-written texts from different distributions. What methods could improve generalization to different domains of human-generated text?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers unanimously voted to reject this paper, and I tend to agree with the reviewers here.  The reviewers point out that the contributions here are modest, the experiments are narrow, the presentation is lacking, among other criticisms.  The reviewers do believe this paper has the potential to be impactful, so I encourage the authors to improve their work, but I recommend rejection for now.