A Survey on the Possibilities & Impossibilities of AI-generated Text Detection
Authors: Soumya Suvra Ghosal
Souradip Chakraborty
Jonas Geiping
Furong Huang
Dinesh Manocha
Amrit Bedi
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have revolutionized the domain of natural language processing (NLP) with remarkable capabilities of generating human-like text responses. However, despite these advancements, several works in the existing literature have raised serious concerns about the potential misuse of LLMs such as spreading misinformation, generating fake news, plagiarism in academia, and contaminating the web. To address these concerns, a consensus among the research community is to develop algorithmic solutions to detect AI-generated text. The basic idea is that whenever we can tell if the given text is either written by a human or an AI, we can utilize this information to address the above-mentioned concerns. To that end, a plethora of detection frameworks have been proposed, highlighting the possibilities of AI-generated text detection. But in parallel to the development of detection frameworks, researchers have also concentrated on designing strategies to elude detection, i.e., focusing on the impossibilities of AI-generated text detection. This is a crucial step in order to make sure the detection frameworks are robust enough and it is not too easy to fool a detector. Despite the huge interest and the flurry of research in this domain, the community currently lacks a comprehensive analysis of recent developments. In this survey, we aim to provide a concise categorization and overview of current work encompassing both the prospects and the limitations of AI-generated text detection.  To enrich the collective knowledge, we engage in an exhaustive discussion on critical and challenging open questions related to ongoing research on AI-generated text detection.

Published in Transactions on Machine Learning Research (01/2024)
A Survey on the Possibilities & Impossibilities of Al-generated Text Detection
Soumya Suvra Ghosal*
sghosal@umd.edd edu
University of Moryland, College Park, MD, USA
Souradip Chakraborty*
schakra330umd edu
University of Moryland, College Park, MD, USA
Jonas Geiping
jgeiping@umd edu
University of Moryland, College Park, MD, USA
Furong Huang
furongh@umd. edu
University of Moryland, College Park, MD, USA
Dinesh Manocha
dmanocha@umd.ccm
University of Moryland, College Park, MD, USA
Amrit Singh Bedi
amritbdd@mm. edu
University of Moryland, College Park, MD, USA
Abstract
Large language models (LLMs) have revolutionized the domain of natural language process- ing (NLP) with remarkable capabilities for generating human-like text responses. However, despite these advancements, several works in the existing literature have raised serious con- cerns about the potential misuse of LLMs such as spreading misinformation, generating fake news, plagiarism in academia, and contaminating the web. To address these concerns, a consensus among the research community is to develop algorithmic solutions to detect Al- generated text. The basic idea is that whenever we can tell if the given text is cither written by human an AI, cerns. To that end, plethora of detection frameworks have been proposed, highlighting the possibilities of Al-generated text detection. But in parallel to the development of detection frameworks, researchers have also concentrated on designing strategies to elude detection, ie., focusing on the impossiblities of AT-generated text detection. This a crucial step in order to make sure the detection frameworks are robust enough and it is not too easy to fool detector. Despite the huge interest and the fiurry of research in this domain, the community currently lacks comprehensive analysis of recent developments. In this survey, aim to provide concise categorization and overview of current work encompassing both the prospects and the linitations of Al-generated lext detection. To enrich the collective knowledge, we engage in an exhaustive discussion on critical and challenging open questions related to ongoing research on AT-generated text detection.
utilize this information to address the above-mentioned con-

---

Published in Transactions on Machine Learning Research (01/2024)
Contents
1 Introduction
3
1.1 Our Focus
4
2 Preliminaries: Al-generated Text Detection
4
3 Thcoretical Insights of AI-Generated Text Detection
6
4 Towards the Possibilities of Al-generated Text Detection
8
4.1
Prepared Detectors
8
4.1.1
Overview of Methods based on Watermarking
4.1.2
Biased-Sampler Watermarks.
9
4.1.3
Pseudo-Random Watermarks.
12
4.1.4
Retrieval Based Methods
13
4.2
Post-hoc Detectors
................................ 14
4.2.1
Zero-shot Detection
4.2.2
Methods based on Training and Finetuning of Classifiers
16
Towards the Impossíbilities of Al-generated Text Detection
19
5.1
Paraphrasing Based Attacks
19
5.1.1
Red Teaming Language Model Detectors with Language Models (Shi et al., 2023)
19
5.1.2
Paraphrasing Evades Detectors of Al-generated Text (Krishna et al., 2023).
19
5.1.3
Paraphrasing by Humans
5.2
Generative Attacks
20
5.3
Copy-Paste Attacks
20
5.4
Spoofing Attacks
21
6
Open Questions
21
6.1
Regulating Large Language Model Watermarks.
21
6.2 Accurate Evaluation of Detectors
................................
22
6.3
Conditions for Al-generated text. Delection
................................
22
6.4
Improving the Fairness of Text Detectors.
22
................................
6.5 Improving the Robustness of Text Detectors.
23
6.6 Discussion on Different Detection Scenarios in Practice
23
6.7 Can We Leverage Our Knowledge of Input Prompts for Better Detection?
Dete.oo.........................
25
6.8
Is Semantic Watermarking Possible?
25
7
Conclusions
25
2

---

Published in Transactions on Machine Learning Research (01/2024)
1 Introduction
In recent years, the natural language processing (NLP) community has witnessed a paradignn shift with the introduction of Large Language Models (LLMs) (Devlin et al., 2018; Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2022). Recently, ChatGPT (OpenAI, 2022), a chatbot based on the GPT-3 (Brown et al., 2020) model architecture, has attracted the attention of millions of users with its remarkable capability of generating coherent responses to a variety of queries. However, these advancements come as a double-edged sword to society. Specifically, the ease of using LLMs has worried the community about their potential misuse. For example, recent research has shown that LLMs can be used to generate fake news, spread misinformation, contaminate the web, or engage in academic dishooessy (Stokel-Walker, Walker, 2022). Additionally, L.Mss could also be utilized for plagiarism, intellectual property theft, or the generation of fake product reviews, thereby misleading consumers and negatively impacting businesses (Chakraborty et al., 2023). Further, a recent study by Carlini et al. (2023) highlighted the risks and challenges related to the generation of synthetic training data using LLMs. Hence, although LLMs have led to a remarkable development such as language translation (Vaswani et al., 2013; Huck et al., 2018; Radford et al., 2019; Brown et al., 2020), question-answering (Guu et al., 2020; Xiong et al., 2020), and text classification (Howard & Ruder, 2018). mitigating their potential misuse is the need of the hour (Brown et al., 2020; Tamkin et al., 2021). Henco, a crucial question to address is: " "How can we harness the benefits of LLMs while also mitigating their polential drawbacks?"
several tasks
Even though it is difficult to answer the above question completely, a popular suggestion within the research community to address these cthical concerns is to accurately detect Al-generated text and separate it from human writings, thereby ensuring the responsible deployment of LLMs. Such capability for AT detection would reduce the potential for misuse. Whether detection can effectively reduce harm is situation-specifee. In some cases, even a limited capability to detect Al-generated text effectively combats misuse. For example, it can prevent accidental misattribution or make it more difficult to mislead consumers. In other situations, such as targeted fake news campaigns by motivated actors, only a strong capability to detect A Al-generated text, would fully prevent misuse.
However, detecting Al-generated text is challenging problem to solve in general. A study by Gehrmann et al. (2019) has shown that the ability of humans to detect AT-generated text was, unfortunately, only slightly better than random classifier, even when tested against the language models available back in 2019. In light of this, researchers turned their attention to developing automated systems that can detect Al-generated text based on features that may not be easily recognizable by humans. Given the gravity of the problem, several works in the past few years have focused on the problem of AT-generated text dctection. Common approaches include leveraging watermarking techniques (Aaronson, 2023; Kirchenbauer et al., 2023a; Zhao et al., 2023a; Kuditipudi et al., 2023b), designing detectors based on statistical metrics (Gehrmann et al., 2019; Mitchell et al., 2023) or fine-tuning classifiers on a set of training samples (Solaiman et al., 2019; Chen et al., 2023; Wu et al., 2023a). In contrast to the development of detection frameworks, interestingly, recent studies (Sadasivan et al., 2023; Krishna et. al., 2023) have shown that state-of-art detection frameworks can be vulnerable and fragile to paraphrasing- based attacks. Further, Sadasivan et al. (2023) have also provided interesting theoretical insights about the impossibility of Al-generated text detection, outlining the fundamental impossibility of detecting a theoretically optimal language model. This analysis is followed by work elucidating the possibilities of AT-generated text detection in Chakraborty et al. (2023), which highlights that it should always be possible to detect Al-generated text via increasing the text length unless the distributions of human and machine-generater language model is theoretically optimal. This possibility result is also supported in subsequent work in Kirchenbaucr et al. (2023b), showing that for the example of watermarking, empirically, attacks such as paraphrasing only dilute the signal that the text is Al-generated and that observing a sufficiently large amount of text makes detection possible again. However, how much text is available from a single source is application-specffi.. In this review, we provide a taxonomy of recent studies of this nature, discuss their interplay, detail their approaches to detection, outline their inherent limits, and identify the applications where specific techniques show promise.
texts are the same over the entire support, i... the
3

---

Published in Transactions on Machine Learning Research (01/2024)
1.1 Our Focus
Based on the recent research locus to address the problem of Al-generated text detection (AI-GTD), in this survey, we aim to provide quick description of such recent results. We have divided the existing literature into two parts; towards the possibility and towards the impossibility of AL-GTD. The two categories are detailed as follows.
Towards the Possibilities of AI-GTD (cf. Sec. 4): Under this category, we specifically review notable, recent frameworks designed detect Al-generated text. These frameworks are categorized into two major groups based on usability: prepored (Sec. 4.1) and post-hoc (Sec. 4.2). We further classify prepared detectors into two main sub-groups based the working principle: Watermarking- based (Sec. 4.1.1) and Retrieval-based (Sec. 4.1.4). Based on the sampling principle, watermarking detectors can be further granularized into biased-sampler watermarks and pseudo-random water- marks. On the other side, post-hoc detectors primarily encompass Zero-shot detection (Sec. 4.2.1) and detection based on fine-tuning classifiers (Sec. 4.2.2).
Towards the Impossibilitics of AI-GTD (cf. Sec. 5): Under this category, we review different attack schemes designed to evade detection. For the ease of readers, we subdivide the attacks into four major categories: Paraphrasing based attacks 5.1, Copy-paste attacks 5.3, Generative attacks 5.2 and Spoofing attacks 5.4.
Figure 1 provides a pictorial representation of the nlove-mentioned categorization. In this survey, aim to provide a detailed description of each of the results and also discuss some interesting open questions at the end. We start by providing a brief description of the functionality of language models and AI-GTD in Sec. 2. Then we proceed by reviewing the recent. theoretical insights regarding Al-GTD explored by Chakraborly et al. (2023); Sadasivan et al. (2023) in Sec. 3. Next, we provide a comprehensive investigation of different studics highlighting the avenues and limitations of AI-GTD in Sec. 4 and Sec. 5, respectively. Finally, in Sec. 6, we provide a detailed systematic discussion on various critical questions related to the regulation of language models, designing robust test statistics, and accurate characterization of human distribution. Further, based on our discussion, also emphasize future works to focus designing fair and robust detectors immune to paraphrasing attacks.
2 Preliminaries: Al-generated Text Detection
In this section, we define the mathemattcal nolation for the language model, and try to define the problem of AI-GTD concisely as follows.
Language Models (LM). Let first define language model mathematically. First, let us denote a vocabulary set as V and we denote language model by mapping Le (parameterized by 0). A language model Co takes a sequence of tokens (called prompt) as input denoted by h := (2,, ,NJ, where each token h; E V. The prompt is fed as input to the language model and it generates or predicts the first output token 8p E V. To determine the next token 81, the input prompt h and generated token 8p are again fed as input to the language model as a new prompt [h, sol. The process is reiterated for all time steps. Let the input prompt for the t-token be given by [h, s1:-1] i] where S {sp, S1. sI.... st-1). In order to generate the next token for the given prompt, the LM outputs a [V]-dimensional logit vector la corresponding every token in vocabulary V such that l, Cohh,, s1.)-)). Then, softmax function is used to map the logits L, into a probability distribution over the vocabulary set V which denote by Pc, (s, -Ih, I)). Finally, the next token 8t is generated by sampling from the distribution Pt- Some commonly used sampling techniques in the literature include Greedy Sampling, Random Sampling, Beam Search (Freitag & Al-Onaizan, 2017), Top-p sampling (Holtzman et al., 2019) and Top-k Sampling (Fan et al., 2018).
The probability of sampling token 8; E V is given by (i( Evey
explex(s)
Al-generated Text Detection. In NLP, text detection problem has been commonly treated as a binary classification task (Jawahar et al., 2020). The main objective to classify whether candidate passage is
4

---

Published in Transactions on Machine Learning Research (01/2024)
Taxonomy of AI-GTD
Detection Schemes
Attack Schemes (Towards Impossibilitt)
(Towards Possibility)
Prepared Detectors
Detectors
Paraphrasing Attacks
Watermarking Based Detection
Zero-shot Based Detection
Paraphrasing model
Copy-Paste Pasto Attacks
GLTR (Golrmann at at, 2019)
Red Teamiin Language Model Detectors with LMs 2023)
On the Reliabiltt of Watermarks for LLMs KKrchenbauer 2023b)
Bleeed Sampler
Watermark
DetectOPT (Micchel 2023)
Paraphrasing detectors Al-GTD (Knishna at al,, 2023)
Provable Robust Watermarking for AI-
(Yóng et, 2029)
Generative Attacks
12hao et al, 2023e)
Smaller Language Models are Better Black- box Machine- Generated Text Detectors (Mtrecighaalan
the Reliability Watermarks LLMS
On the Reliability of Watermarks for LLMs
Kirchenbsuer at, 2023b)
KKrchenbauer 2023b)
Paraphrasing by human
Natural language watermarking via paraphraser-beee lexical substitutton (Gleng 2023)
On the Possibilities of Generated Text Detection (Chakreborly byboryyyy al, 2023)
Reliability Wittrrmaris LLMs
Spoofing Attacks
OKrchenbsuer ot at. 2023b)
Pseudo-Randee
Can Al-Generated text reliably detected?
Aaronson Watermark (Aaronson, 2023)
(Sadasivan at, 2023)
Training/Fine-tuningg Classifier
Undetectable Watermarks for LM (Christ at, 2023)
GPT-Sentinal [Chen
LLMDet: A LLMS Detection Tool IWu ot at, 20231
Close 2023)
Retrieval-Basee Based
Ghostbuster: Detecting Text Ghosteritten by LLMs (Verm 2023)
Robust Al-Text wia Adversarial Lewning (HU 2023)
Paraphrasing evades detectors of Al-OTO
(Knshna et 2023i
Figure 1: Categorization of all explained methods. In this survey, have divided the existing literature into parts: towards the possibility and towards the impossibility of AL-GTTD. As explained in Section 1, based on usability all detector frameworks
be categorized into major groups: Prepared (Sec. 1.1) and Post-hoc (Sec. 1.2). Prepared frameworks can be further sub-categorized into four major groups based on the working principle: Watermarking: based detection (Sec. 4.1.1) and Retrieval-bar baasee detection (Sec. 4.1.4). Based the sampling principle, watermarking detectors can be further sub-gramularized into Biased Sampler watermarks and Paeudo-random watermarks. On the other branch, Post-hoc detectors primarily encompass Zero-shot detection (Sec. 4.2.1), detection based on fine- tuning classifiers (Sec. 4.2.2). For impossibilities, we review different attack schemes designed to evade detection We subdivide the attacks three major categories: Paraphrasing based attacks 5. Copy-paste attacks 5.3, Generative attacks 5.2 and Spoofing attacks 5.4.
5

---

Published in Transactions on Machine Learning Research (01/2024)
News Articles
Scientific Content
Lust
Literature
Social Media
X
f
D(s) < 8
Other Sources
Input Sample (s)
'The NASA independent study team not find evidence that unidentified aerial phenomena have an extraterrestrial origin..."
Text Detector D(s)
Bichine-Generated
D(s) 2 8
Text Detection Framework. This figure shows a flowchart adapted from Mitchell et al. (2023) to depict the general framework of Al-generated text detection.
Figure 2:
Al-generated or human-written. Let us represent the set of all possible sequences as S. Given an input sequence 8 E S, a text detector D(s) S + R outputs a scalar score, which is then compared against a threshold å to perform detection. score higher than the threshold D(s) 2 serves as an indicator of Al-gencratod content, and conversely. A conceptual representation of the overarching architecture governing the identification of machine-generated text portrayed in Figure
3 Theoretical Insights of Al-Generated Text Detection
Impossibility Result. a recent study, Sadasivan et al. (2023) argues that the current state-of-art Al-generated text dotectors based on watermarking schemes (Kirchenbaner et al., 2023a) and zero-shot clas- sifiers (Mitchell et al., 2023) are not reliable in practical scenarios. To support their argument, they show that simple lightweight neural-network-basee paraphraser such as PEGASUS (Zhang et al., 2020) can sig- nificantly reduce the detection accuracy of watermarking-baas text detectors (Kirchenbaucr et al., 2023a) without causing drastic change in the perplexity score. Further, authors observed 71% degradation in accuracy for zero-shot classifiers such as DetectGPT (Mitchell et al., 2023) when using T5-based para- phraser. These observations highlight that even state-of-art text detectors are not immune to paraphrasing attacks, raising questions about their reliability.
To provide a thcoretical understanding, Sadasivan al. (2023) derived an impossibility result of Al-text detection. It states that when there is a strong overlap between the text distributions generated by humans and the model, even the best text detector performs marginally better than random classifier. Formally, let M and H represent the text distributions of model and human. The area under the ROC (AUROC) curve of a post-hoc detector (cf. Sec. 4.2) 'D can be bounded as:
1 AUROC(D) < (V(,, TV(,
(1)
where TV indicates the total variation distance between the two distributions defined as: TV,H, = maxg P (s E] PP E E]], for any event E. Now from the bound, when there is a strong overlap between machine (M) and human (H) generated text distributions, TVM, H) 0. In this case, the upper bound the AUROC of the detector approaches 1/2, representing random classifier.

---

Published in Transactions on Machine Learning Research (01/2024)
Watermarking solves this problem, as it can synthetically introduce distribution shift between the two distributions. However, detection can still be made difficult by paraphrasing the text (Sadasivan et al., 2023; Krishna et al., 2023). Let M represent the distribution of the paraphrased samples. Then, for any watermarking scheme W, Sadasivan et al. (2023) provide the following bound:
P (s watermarked using W| - PawH(s watermarked using W]+ W 7))
(2)
If the paraphrased distribution M closely resembles the human text distribution H. then TV(M", H) will be minimal. From the bound in Equation 2, this implies: (1) Ps watermarked using W] is high indicating an increase in false positives, i.c., misclassifying human-written text as AI-gencrated, or (2) P=-.M- (s watermarked using W] is low indicating increase in misclassification of watermarked text as human- authored.
Hidden Possibilitics of Text Detection. Interestingly, Chakraborty al. (2023) argue that the impos- sibility results posited by Sadasivan et al. (2023) could be conservative in practical scenarios. Specifically, the impossibility result does not consider the potential number of text sequences available (the possibility of collecting multiple sequences) or the length of the text sequence, which can impact the detectability sig- nificantly, leading to a pessimistic conclusion. Additionally, the assumption in Sadasivan et al. (2023) that all human writings follow a "monolithic" language distribution is not realizable in practice. For example, Chakraborty et al. (2023) provides a motivating example in the critical case of detecting whether a Twitter account is AT-bot or human, and it's natural that in those scenarios, one can expect multiple samples to improve the detection performance. Chakraborty et al. (2023) derive a precise sample-complexity bound for detecting Al-generated text for both IID and non-IID scttings, which highlights the hidden possibility in Al Al-generated lext detection.
IID Setting: For the i.i.d scenario, given collection of independent and identically distributed (i.i.) text examples, {si) they show that the AUROC can be bounded as:
1 AUROC(D) < + TV(M H000) ( - TVMM 8M HH0ny
(3)
2
where, M and H denotes the product distribution of model and human respectively. According to large deviation theory, the total variation distance (TV(M Hd²) approaches 1 at rate exponential exp(nIa(M, H. + o(n)), where IcMM, H) represents the
to the number of samples: TVMM H(*) Chernoff information. Now, with increase in the number of samples: n co, the total variation distance TVMM"n h²* approaches I exponentially fast thereby increasing the upper bound on AUROC. Thus, increasing the number of samples significantly improves the possibilities of Al-generated text detection. The sample complexity bound in Chakraborty et al. (2023) states that when the model and human distribution are y close to each other (TV H) = 7), to achieve an AUROC of e, the number of samples required are:
n n= S2 ( log
1 (-=
The bound in Equation 4 indicates that the number of samples required to perform detection increases exponentially with an increase in the overlap between machine and human distribution. This observation has been further empirically highlighted in Chakraborty et al. (2023) (Figure 1).
Non-IID Setting: Similarly, Chakraborty al. (2023) also extended the results to non-iid scenarios by showing that if human and machine distributions are close TV(m, h) ? > O, then to achieve an AUROC of e, it requires
1 log (1-2)+ + Σ (o 1)p; + 82 (
log ( 1
L Σ (cj - 1)p; j=1
n = Ω
72 1 (1-)
(5)
number of samples for the bost possible detector, for any c E [0.5, 1). where, as before n represents the number of (dependent in this case) samples/sequence and L represents the number of independent subsets where each subset. is represented by tj with ej samples Vj E (1,2..., T.). I.). The non-iid result. in (Chakraborty
7

---

Published in Transactions on Machine Learning Research (01/2024)
et al., 2023) is more general and can be analyzed under different settings by varying the mumber of subsets with varying samples in each subset. The characterization of sample complexity results with Chernoff information as derived in Chakraborty et al. (2023) is novel and brings several new insights on how to design better detectors and watermarks.
To summarize, analysis in Sadasivan et al. (2023) highlights that when the model and human distribution overlap or are close to each other, i.e, when the total variation distance between the distributions is low, detection performance can only be random. However, when the number or length of samples available detection time is taken into consideration, the analysis in Chakraborty et al. (2023) finds that for any level of closeness, a number of samples exist that provide strong detection performance. The theoretical claims were validated in (Chakraborty et al., 2023) for real datasets Xsum, Squad, IMDb, and Fake News dataset with seate-of-the-art generators and detectors. The claims have been further validated also in the context of reliable watermarking in Kirchenbauer et al. (2023b). However, Sadasivan et al. (2023) highlights certain assumptions in these prior works that might not be practical for the detection scenarios. For example, improved detection performance with multiple iid samples as shown in the first part of work by Chakraborty et al. (2023) might not be practical in several scenarios like evaluation in exams, etc. Similarly, the defense by Krishna et al. (2023) requires storing the outputs of the LLM in a database, which, although efficient, might lead to privacy concerns. Additionally, with the possibility of spoofing and generative attacks in several real-world scenarios, it will be even harder to perform detection due to reduced TV distance.
Hence, we note that the claim of possibility or impossibility requires additional context about the problem, which is extremely important. Thus, a detailed description of various problem instances and detectability is presented in Sec. 6.6, where we discuss the detectability across different practical problem instances.
4 Towards the Possibilities of Al-generated Text Detection
In the light of ownership and practical usability, Al-generated text detectors can be categorized into two distinct groups: "Prepared" and "Post-hoc" detectors. Prepared detection scheme primarily include water- marking and retrieval-based detectors, which involves proactive involvement of the model proprietor during the text generation process. In contrast, Post-hoc detectors encompass zero-shot detection techniques or fine-tuned classifiers which can be used by external parties.
4.1
Prepared Detectors
4.1.1 Overview of Methods based on Watermarking
Although watormarking has been a long-known concept in the literature for hiding information within data, implementation was mostly challenging in the early days due to its discrete nature (Katzenbeisser & Petit- colas, 2016). Synonym substitution (Topkara et al., 2006), synthetic structure restructuring (Atallah et al., 2001), and paraphrasing (Atallah et al., 2001; 2002) were the commonly used approaches in the past for em- bedding a watermark into existing text (Zhao et al., 2023a). However, with advancement in neural language models (Vaswani et al., 2017; 7; Devlin et al., 2018), rule-based approaches have been replaced with improved techniques based on using mask- infilling models (Ueoka et al., 2021). Recent approaches to watermarking include learning end-to-end models (Ziegler et al., 2019; Dai & Cai, 2019; He et al., 2022a;b) with both encoding and decoding of each sample. Watermarks specifically designed for large language models have recently gained major attention as a technique to detoct machine-generated text (Aaronson, 2023; Kirchen- bauer et al., 2023a; Zhao et al., 2023a; Christ et al., 2023a; Zhao al., 2023b; Kirchenbauer et al., 2023b; Liu et al., 2023; Kuditipudi et al, 2023a; Zhang et al., 2023).
Formally, a general watermarking scheme can be defined as consisting of two probabilistic polynomial-time algorithrss Watermark and Detect (Zhao et al., 2023a). The Watermark algorithm takes in a language model L as input and modifies the model's outputs to encode a signal into generated text.
During inference, given text sequence s and detection key k, the Detect algorithm outputs if s was generated by 2 or 0 if it is generated by any other model. Ideally, a watermarking- based detector should exhibit the following properties: (1) Preserve the original text distribution, (2) Be detectable without access
8

---

Published in Transactions on Machine Learning Research (01/2024)
to the language model, and (3) Be robust under perturbations and distribution shifts (Kuditipudi et al., 2023a). Christ et al. (2023a) also highlights that (4) the watermark should be ideally undetectable for any party not in control of the secret key that defines the watermark to make the detection widely and easily applicable.
Based on the sampling principle, watermarking schemes can be further sub-classified as either "biased- samplers", where the watermarked model's C output distribution is modified probability distribution p. = Pr (s, = (1+,,) on the vocabulary set V, or as "pseudo-random watermarks, where the model's output probability is formally not modified, but instead collapsed onto particular sample from Pc.. chosen pscudo-randomly based on local context.
We describe a number of watermarks from the first category in subsection 4.1.2, and of the second calegory in subsection 4.1.3.
4.1.2 Biased- Sampler Watermarks.
Biased-Sampler watermarking (Kirchenbauer et al., 2023a;b; Zhao et al., 2023a) modifies the token distribu- tion at each time step to encourage the sampling of tokens from a pre-determined category (a green list at each token). As an advantage, since biased sampling creates a generic shift in the probability distribution, it can be coupled with any sampling scheme. However, biasing the output distribution can also lead to increased perplexity and degradation in text quality. Below, we summarize a few notable studies leveraging biasodl-smmple watermarking scheme to detect AT-generated text.
A Watermark for Large Language Models (Kirchenbauer et al., 2023a) Kirchenbauer et al. (2023a) was the first to propose a watermark based on biased sampling from large language models and derive a matching detector for Al-generated text based on statistical metrics. Note that, watermarking language models require access to the generated probability distribution over the vocabulary at each time step. Let S1 = {51, $2,-.., ,t-13 be the output of the language model till t- 1-th step and le E R be the logit scores generated at the t-th step. Now, the watermarking operation can be described in the following steps:
First, the tokens in the vocabulary set V is divided into two disjoint subsets namely red and green list. This membership labeling is controlled by a context-dependent pseudo-random seed generated by hashing the set of tokens in the last e time steps {st-es 8t st-1), where e is the context width. In this paper, the context width is mainly set c = 1, i.e., only the token in the previous time step st-1 is used for modulating the seed. Based this division, the green list consists of 7/V] tokens where E (0, 1) represents the ratio of tokens in green list to red list.
2. For sampling St, Kirchenbauer et al. (2023a) discusses two major schemes to encode a watermark based on these green lists.: (a) "Hard" watermarking, and (b) "Soft" watermarking. Hard water- marking is designed to always sample a token from the green list and never generate any token from the red list. A major drawback is that for low entropy sequences where the next token is almost deterministic, hard watermarking may prevent the language model from producing them, resulting in degradation of the quality of watermarked text. To this end, Kirchenbauer et al. (2023a) proposes a "softer" version as their main watermarking strategy, in which for every token belonging to the green list, a scalar constant (a) is added to the logit scores:
& = [ + a I[u E green]
(6)
where I indicates an indicator function. This helps in creating a bias towards sampling tokens from green lists more frequently as compared to red lists. After modification, the logit scores are generally passed through a softmax layer and the token 8t is sampled from the modified distribution with any desired sampling scheme.
Given a text passage, to chock for a watermark one does not need access to the source model as the random seed can be re-generated from the previous tokens. To identify whether text sequence is watermarked
9

---

Published in Transactions on Machine Learning Research (01/2024)
not, one needs to simply recompute the context-dependent pseudo-random seeds and count the number of green tokens. Let {81, $2,".. .8N represent test sequence of length N. Then the number of green tokens in s is calculated as |sal = | E green] where I represent the indicator function. The detection is then carried out by evaluating the null hypothesis which assumes that the sequence is generated by a natural writer with no knowledge of the green or red list. To lest the null hypothesis, 2 z-score is calculated as:
2 = (Iscl - /N) /NN/N/N
(7)
where Y represents the green list ratio. A text passage is classified as "watermarked" if 2 2 6, where 6 is the classification threshold.
When the null hypothesis is true, i.e, the text sequence is generated by natural writer, |sgl (number of green tokens) follows a binomial distribution with a mean of N and variance of N This is because writings 2 without knowledge of the watermarking scheme would has a 7 (e.g. 50%) probability of each token being sampled from a green and 1 y from a red list. With an increase in the mumber of tokens, based on Central Limit Theorem, 8G can be approximated with Gaussian distribution and thus can be approximated with 2 scorc. Human-generated text without the knowledge of the green list rule is expected to have lower number of green tokens and thus lower z-score as compared to machine-generater text. For shorter texts the Gaussian assumption does not hold anymore, however, Fernandez et al. (2023) has shown that detection is still feasible using an actual test of the true binomial distribution. Kirchenbauer et al. (2023a) and Fernandez et al. (2023) also both point out that in case a document consists of repetitive texts, the detection scores defined above might be erroneous. This is because the independence criterion necessary for calculating the z-scores does not hold in self-similar texts, with many repeating contexts, thereby artificially modifying the score. As solution, Fernandez et al.; Kirchenbauer et al. propose scoring only those tokens during detection for which the watermark context. plus the token itself ({st-e;"" , 8t) have not been previously seen and Fernandez et al. (2023) show that this modification, in tandem with testing against binomial distribution, leads to analytic false positive rates that closely match empirical estimates.
Provable Robust Watermarking for AI-Gencrated Text (Zhao et al., 2023a) Recent studies by (Sadasivan et al., 2023; Krishna et al., 2023) have questioned the robustness of watermarking- based detection models against modifications of the text. In light of that, Zhao et al. (2023a) proposed a robustified watermark-bass framework named GPTWatermark. GPTWatermark similar to the watermarking scheme introduced in Kirchenbauer et al. (2023a), except that for every token they use a fixed split controlling the green and red list, i.e. context width of 0, whereas in Kirchenbauer et al. (2023a), the split is controlled by pscudo-random seed based on the hash of the previous tokens.
A fixed split as in Zhao et al. (2023a) is optimal in terms of robustness to edits of the text. This robustness comes with a trade-off in detectability. As pointed out in Christ et al. (2023a), the length of the context (and entropy of the sequence) determine how detectable a watermark signal is, i.c how likely an attacker is to recover the watermark from observing watermarked text. Detectability also implies that impact on text generation quality or impact on users (who might observe that the model is less likely to use certain words) cannot be ruled out, but Zhao et al. (2023a) show that that these effects are likely small in practice.
As an improvement over Kirchenbauer et al. (2023a), Zhao el al. also provides a theoretical understanding of the robustness properties of the watermarking scheme. As a threat model, the paper considers adversaries with only black-box access to the language model. Given an adversary A and a watermarked sequence s = (s1, $2. .......... ,.. s,). let the adversarially modified output be given as SAA Zhao et al. assume that the edit distance between s and 8Λ is upper bounded by some constant v) such that ED(s, S.)) < r.. Where ED represents the edit distance and is defined as the number of operations required to transform s into S,A Then the maximum text edit distance required to evade the GPTWatermark can be given by:
Vnl 2e 5) n
7/a 1
2
0<
1 ++3 7
2
where 2 represent the z-score corresponding to the watermarked sequence s, 7 represents the green list ratio, and 8 represents the classification threshold of the detector. Similarly, the maximum edit distance required
10

---

Published in Transactions on Machine Learning Research (01/2024)
to evade the baseline watermarking scheme introduced in Kirchenbauer et al. (2023a) detector can be given by:
Vn( - 6 77 = 2 + 2
TVn
2x
<
2
when setting a context width of 1. The above expressions highlight that it takes twice as many edits to evade GPTWatermark as compared to Kirchenbauer al. (2023a). Note that a number of attacks, such as paraphrasing and generative attacks, do not assume that an attacker would be restricted to make only nj edits to the text.
On the Reliability of Watermarks for Large Language Models (Kirchenbauer et al., 2023b) As a refinement of the baseline watermarking scheme introduced in Kirchenbauer et al. (2023a), in this paper, Kirchenbauer et al. (2023b) proposed a more robust and improved hashing scheme for generating watermarked lext. Recall that Kirchenbauer el al. (2023a), the hashing scheme focuses context width of e = L, i.e, for the t-th time step the random seed for splitting the vocabulary is generated by hashing the token at only the t - 1-th step. Let this hashing scheme be denoted as LeftHash. An alternative hashing scheme (denoted as SelfHash) also uses the token at the position t itself in addition to the tokens the left of t. Both scheme can then be used with arbitrary context width c. The analysis in Kirchenbauer et al. (2023b) suggests using smaller context widths provides more robustness to paraphrasing attacks, but related to findings in Zhao et al. (2023a) and Christ et al. (2023a), a larger context width leads to a less detectable watermark, which comes with a smaller impact on text quality and a robustness against attacks that attempt to reverso-engineer the watermark.
Formally, hash function is defined as f : N - N, which takes in set of tokens {s 8 as input and outputs pseudo-random number. Let N be secret salt value and P : N + N be an integer pscudo-random function. In this paper, Kirchenbauer et al. introduces the following hashing schemes:
1. Additive: This is improved version of the hashing scheme introduced in Kirchenbaner et al. (2023a). Specifically, the context width used for hashing is increased. The hashing function is defined as f(s) P(a Eiz s:). This form of hashing is not robust to attacks that can remove a token s; from the context.
2. Skip: This scheme of hashing only takes into account the leftmost token in the context: f(s) = P(as.).
3. Min: For this scheme, as the name suggests, the hash function is defined as the minimum of the hash value generated using each token s; in the context: f(s) = minge P(as;)
Empirical study by Kirchenbaner et al. (2023b) shows that for larger context widths Min and Skip variants are much more robust to paraphrasing attacks as compared to the Additive hashing scheme. However, in terms of the diversity of the text generated, the Additive scheme ranks better as compared to other variants, line with Christ al. (2023a).
Further, Kirchenbauer et al. (2023b) also discuss the question of detecting watermarked text embedded inside a much larger non-waternarked passage. There, computing the z-score globally would not provide an accurate measure. In light of this, Kirchenbauer et al. (2023b) propose windowed z-score evaluation for detecting watermarked sequences in long documents. Let s be a text passage of length T consisting of watermarked tokens. The detection is carried out by first generating a binary vector æ E {0, 1)x indicating the membership label of each token. Let P = E 24 be the sum of the binary hits and be the green list fraction. Then the WinMax score is calculated as:
(p) - pp) - 101--/11
ZWinMax max (J
for i <
Va(l - 7 ii)
11

---

Published in Transactions on Machine Learning Research (01/2024)
Natural language watermarking via paraphraser-based lexical substitution (Qiang et al., 2023) Qiang et al. propose a watermarking framework by incorporating paraphrase-based lexical substitution method. The core idea involves generating substitute candidates for each token in a sequence using a paraphraser and embedding them in a way to preserve the meaning. Let 8, represent the i-th token in a sequence s to be encoded using the walermarking framework. Also, let m E {0, 10 be a fixed binary watermark sequence. The watermarking procedure involves the following steps: (1) First, using a pre-trained paraphraser, substitute candidates are generated for the token 8;- Let yi represent the most probable substitute token generated by the paraphraser. (2) Next, the algorithm checks the validity of y substitute token for 8;. If 8; == Para(us), where Para represents the paraphraser, then Vi is considered a valid substitute token. (3) Next, the list C' = {Mi, 8 is sorted in an ascending alphabetical order. Finally, the encoded token (8:) is set using the watermark sequence m as: & = C[m[i]]-
After the watermarked text generated, one can check for the watermark in a given sequence by extracting the binary watermark sequence m using the technique discussed above. Text generated from sources other than the watermarked language model would result in different binary sequence. The paper lacks detailed analysis to understand the effect of paraphrasing attacks on this watermarking scheme.
4.1.3 Rando- Watermarks.
Pseudo-random watermarking schemes (Aaronson, 2023; Christ et al., 2023a; Kuditipudi et al., 2023a) oper- ate by minimizing the distance between the watermarked and original distribution, with the aim of making the watermark both undetectable and unbiased in expectation. Next, we briefly summarize few frameworks leveraging pseudo-random watermarks for AI-GTD.
Aaronson Watermark (Aaronson, 2023). Aaronson proposes a watermarking scheme in which given the tokens generated till t 1-steps S1:-1 {s1, $2, s the token at the t-th step is sampled $ = arg max To pit.s. Here r, E [0, 1] V v E V are secret real numbers generated using a psoudo-random function using a secret key sk, rg = fex (stcc. ... v), Given a test sequence s = (2,, ,Ny,, the presence of watermark identified by thresholding on the detection score, which is calculated as:
d
where r = fat(St-es 8t,,,,,,,))) Fernandez et al. (2023) highlighted an interesting property of this water- marking scheme. In expectation over the randomness of the secret vector r = [ri,""", vil, the probability of 8t being token is exactly pe(w), i.e, E[P(s, = w)] = pr().. It means, that expectation over r, this watermarking scheme does not bias the distribution.
Undetectable Watermarks for Language Models (Christ et al., 2023a) Prior watermarking ap- proaches introduced in Kirchenbaner et al. (2023a;b); Zhao et al. (2023a) operate through altering the distribution of the model output thereby leading to degradation in the quality of text generated (Christ et al., 2023a). Instead, Christ et al. focus on developing a watermarking scheme such that a watermarked text is computationally indistinguishable from non-watermarked text, a property that further implies no degradation in text quality (otherwise the presence of the watermark would be detectable through observa- tion of reduced quality). They formalize the cryptographic notion of a watermark where is computationally intractable distinguish a watermarked text from the original output, and then develop a watermark ful- filling these requirements. Effectively, the watermark of Christ et al. (2023b) is intractable to observe for any parties not in possession of the secret key used to encode it.
Given a vocabulary set V, let V represent the binarized version of the vocabulary set where each token o E V is encoded as a binary vector in {0, Mogg(())) Let F. represent a pseudo-random function modulated by the secret key sk e 10, where À represents the security parameter. Note, that the output of Fs can be interpreted as a real number in [0, 1]. Now, let us consider the process of generating the i-th bit, given the previous bits by bg. bi-1 are already decided. Let (1( represent the probability of the i-th bit being 1 according to the original model. For the i-th bit, the watermarking model outputs b, 1 if
12

---

Published in Transactions on Machine Learning Research (01/2024)
(sb((b1, 02. .b-1). ())))))))......))))))))))))) S (1) otherwise b, O. Note, that the probability of the i-th bit being is exactly (1( since the output of FAx: is drawn uniformly from [0, Thus, the generated watermarked text follows the same distribution as the original text making it computationally indistinguishable.
For detecting the watermark in a given sequence, the secret key sk is leveraged. Let x = f-r,-.. , represent a binary encoded test sequence. For each text bit Fs; the detector calculates a score using the current bit and secret key sk:
In( Fas (Ti; i)) In (a if =0
if 3i
m 8k)
where r; f-1,-.. [4-1). Finally, the score is summed over all the text bits, = (x)==>= m(x,,,8), Unlike watermarked text, where the value of Fax (r..i) i) is correlated with 2,, in non- aotwwatermrrked text (a i) is independent of dy. Thus the expected value of c(x) would be larger in watermarked text as compared to wotemmarrdr text.
Although undetectable, Christ et al. caution that watermarking schemes, in general, cannot be made un- removable and can always be evaded using strong generative attacks that modify the entire text. However, Christ et al. (2023a) do not provide any empirical analysis to understand the actual robustness of the pro- posed undetectable watermarking schemes against, e.g. partial paraphrasing attacks, leaving the empirical robustness to edits uncertain.
4.1.4 Retrieval Based Methods
Paraphrasing Evades Detectors of Al-gencrated Text (Krishna et al., 2023). In a recent study, Krishna et al. show that detection algorithms (Kirchenbauer et al., 2023a; Mitchell et al., 2023) despite impressive performance can be vulnerable against paraphrasing attacks. Hence, to protect against para- phrasing, in this paper Krishna et al. proposes a retrieval-based detection strategy. This defense approach requires a database that store all machine- generated text from a particular model, and as such also a de- fense that can only be mounted by the model owner. Specifically, given input prompt and corresponding output generated by the language model, the API needs to store both the prompt and output sequence in database.
During inference, given a text sequence, similarity scores are calculated between the samples stored in the database and the given sequence. A high similarity indicates that the given sequence is generated by the same language model. Intuitively, human writings are more likely achieve a lower similarity score as compared machine--eneraaee text (considering that the source and target models are the same). In general, this kind of retrieval-baan detection framework is reliable and to some extent robust to possible shifts in the text caused by paraphrasing attacks (Kirchenbauer et al., 2023b), text passages can be matched based semantic features. This scheme can also be attacked and in recent work, Sadasivan et al. (2023) show that, at a fixed detection length, five rounds of recursive paraphrasing can cause a 75% drop in the detection accuracy of retrieval- based methods. However, it remains unclear whether the failure of a semantic match means that the recursive paraphrasing has modified the text too strongly and removed its original meaning, or whether the text functionally the same, and paraphrasod in a way that the semantic matcher does not pick up on. More research is necessary in this direction to accurately characterize the robustness of retrieval-based detection.
Another question with retrieval-based detection are concerns about privacy. The use of this detection approach is not always realizable, as local limitations and regulations, such as GDPR, might limit the amount of data that can be stored by model companies (Krishna et al., 2023). Nevertheless, retrieval-basee detection could be the most accurate detection approach jurisdictions where it fcasible, and could also be employed during judicial proccedings, when parties may request access the database from model companies.
13

---

Published in Transactions on Machine Learning Research (01/2024)
4.2 Post-hoc Detectors
In contrast lo prepared detection methods are approaches that can detect Al-generated text without prepa- ration or even cooperation by the model owner. This tasks is generally believed to be harder than prepared detection, but much more broadly applicable. Especially in scenarios where a large number of freely avail- able language models are used by independent or uncooperative actors, post-hoc detection is the only way to detect AT-generated text.
4.2.1 Zero-shot Detection
For zero-shol text. detection, no access to machine-generaled hunan-wrillen text samples is required. The core idea is that generic text sequences generated by a language model contain some form of detectable information that can be picked up and flagged by a detector. Detection under this category may be performed using a pre-trained language model which may not be similar to the source model, or through a fully separate statistical approach.
Given a text passage, common approaches for zero-shot text detection include: (1) statistical outlier detec- tion based on entropy (Lavergne et al., 2008), perplexity (Beresneva, 2016; Tian, 2023), or n-gram frequen- cies (Badaskar et al., 2008), and (2) calculating average per-token log probability of the given sequence and then thresholding (Solaiman et al., 2019; Mitchell et al., 2023).
GLTR: Statistical Detection and Visualization of Generated Text (Gehrmann et al., 2019). To aid the detection of machine-generated text, Gehrmann et al. (2019) propose a statistical outlier-based detection framework named GLTR, The framework is based on the assumption that language models generate text by frequently sampling from highly probable words. For the purpose of automated text detection, three tests are undertaken. Specifically, given a sequence s = {51, 82 s,,,, for every token they calculate: (I) the probability of generating the token, (2) the rank of the word in the generated distribution, and (3) the entropy of the generated distribution. A high score in the first two tests indicates that the generated token is sampled from the top of the distribution and low score in the third test mcans given the previous context, the model is highly confident of the generated token. Formally, let the larget language model be defined as Co, the detection scores are defined as:
N N (c)( [cc[s1a-1]
d dprob
(8)
1
drank
rank rank(Pcocuu(/Bsnnll)
(9)
N
-1
dentropy
Peon) log cm
(10)
1EV
where rank(p,b) b) refers to the operation of calculating the rank of token b in any distribution p. The authors empirically show that, unlike language models, human writers tend to use low-probability (defined in Equa- tion 8) and low-ranking words (defined Equation 9) much more frequently. The framework relies on these differences for the detection of machine-generated text.
DetectGPT (Mitchell et al., 2023) Mitchell al. (2023) argue that zero-shot detection frameworks (So- laiman et al., 2019) based on thresholding the log-probability of a text passage, fail to leverage information about the local structure of the probability function. To this end, they propose DetectGPT, zero-shot algo- rithm for detecting machino-generated text. The working principle of the algorithm is based on the empirical observation that lext generaled from LLMs lends to occupy negative curvature regions of the model's 's log probability function. Specifically, given a sequence s and the source language model Co, DetectGPT first gen- crates & perturbations of the given sample (say s, Vi E f1,2,... ,,]) using some perturbation function q. In the paper, perturbations are added using the T5 (Raffel et al., 2020) model. For detecting machine- -ennrreee text, the authors calculate discrepancy metric defined as:
14

---

Published in Transactions on Machine Learning Research (01/2024)
d(x, Cg.g) log Co(s) - E, log Co(s:)
(11)
ō²
where 8² represents the variance of the perturbed text. The paper shows that text generated from language models tends to have a higher discrepancy metric as compared to human-generated text, thereby supporting detection.
A major limitation of DetectGPT is that it assumes white-box access model parameters, which is not always realizable in practice. Further compared to other zero-shot detectors based on statistical outlier detection methods, DetectGPT is far more computationally expensive. Also, Krishna et al. (2023) show that even a single paraphrasing pass with a different model can cause significant degradation in the detection accuracy DetectCPT.
DNA-GPT (Yang et al., 2023) Zero-shot detection frameworks based on thresholding log- probability of given sequence such as DetectGPT (Mitchell al., 2023) suffer from major limitation by assuming white- box access to the model parameters. This means that the detector has access to the generated probability distributions at each time step. However, this is strong assumption as the token probability distributions are not always accessible such in the GPT-3355 model.
In response, Yang et al. (2023) propose a detection strategy for black-box cases where detectors are restricted to only API-level access. Let s = "' represent a given text sequence of length n. For detection, the sentence first truncated into two parts: s(1) = [s1, ,,,1,, and g(2) = [f|m]++ sn], where y is hyper-parameter representing the truncate rate. The core idea of this detection strategy is based on the assumption that the uniqueness of each language model is manifested in its tendency of generating comparable n-grams. Leveraging this assumption, the detection model keeps aside 5(2) and feeds the subsequence ( as input prompt. to the language model with the task of completing the sequence. Multiple outputs are generated based on the imput s(1), Let the set of output sequences be represented as Ω = {s".... (s' ,5*), where K represents the number of output sequences generated. Under the black-box setting, the detection is carried out by comparing the n-gram similarity between the sequences in and s(2). Specifically, the detection score with black-box access is calculated as:
BScore(s, !2 = K
1
K
Σ f(n)
n-grams($*) n-grams(s(2)
g [lllngramm(s()))
k= nmming
where f(n) is weight function for different n-grams. In the paper, the default parameters are set (n) nlog(n), no 4 and N 25. Intuitively, higher detection score indicates that the sequence is machino-generated..
In the white-box setting, where one has access to the generated token probabilities, Yang et al. (2023) leverages the generated probability distribution. Specifically, the detection score with white-box access is caleulated as:
1 K WScorc 22 = K p(s s())
((((2)
log
Smaller Language Models are Better Black-box AI-GTD (Mireshghallah et al., 2023). The authors in Mireshghallah et al. (2023) also target the problem of detecting AT-generated text in black- box cross-model setting where the detector has no access to the source model parameters. For detection purposes, (Mireshghallah et al., 2023) leverage the local optimality test introduced in Mitchell et al. (2023).
The assumption here is that machine- generated texts are likelier to be locally optimal as compared to human- written text. Given a sequence first &o perturbations (s) are generated using a perturbation model such as 75 (Raffel et al., 2020) model. The curvature is then calculated as: d(s) = log (o(8) 1 Σ, log (8:). Note, unlike Mitchell et al. (2023) which assumes white-box access, LG here refers to an unknown language model different from the source model. The paper evaluates a suite of 27 different detector models with varying numbers of parameters ranging from 70M to 6.7B. Based on empirical analysis, they observe two interesting trends:
15

---

Published in Transactions on Machine Learning Research (01/2024)
For cross-model detection, smaller models show stronger performance compared to models with larger capacities. However, they also observe that overlap in architecture family and dataset between the generator and detector model leads to improvement in detection performance.
Interestingly, empirical results highlight that partially trained models are better detectors than fully Irained models. For this experiment, they save checkpoints at different sleps during the training process. They find that the final checkpoint is consistently the worst one in terms of machine- generated text detection. The authors hypothesize that this might be due to overfitting associated with longer training process.
4.2.2 Methods based on Training and Finetuning of Classifiers
Another line of work focuses on training a binary classifier using features extracted from a pre-trained language model for detecting machine-generated text (Fagni et al,, 2021; Bakhtin et al., 2019; Jawahar et al., 2020; Chen et al., 2023). This approach has a longer history with hallmark studies concerning finetuning classifiers for detecting neural disinformation in Hovy (2016); Zellers et al. (2019). In 2019, (Solaiman et al., 2019) achieved a then state-of-art performance by finetuning RoBERTa models for the task of detecting webpages generated by GPT-2. Recently OpenAl's (OpenAI, 2023) work on machine-generated text detection by finctuning a GPT model has followed these development, although OpenAl's detector has now been (aken offline, due to its high false-positive rate. Detectors under this calegory do not require access to model parameters and hence operate under complete black-box settings. However, unlike the zero-shot sctup, supervised training samples are roquired in the form of human and machine-generatee text to train the detector.
Detectors under this category suffer from a few drawbacks: (1) Collecting sufficient data to train the clas- sifier can be challenging, especially in diverse domains where the availability of training samples is a major bottleneck. (2) With recent advancements, text generated by language models has become increasingly similar to human--enerated text, making detection harder (Zhao et al., 2023a)- (3) False-positive rates for these detectors are hard to establish and depend crucially on the data distribution used to train the detector (Liang el al., 2023). Further, recent study by Gambini el al. has shown that detection stralegies designed for smaller models such as GPT-2 lose their efficacy when applied to larger models such as GPT-3. Wolff & Wolff (2020) has also questioned the robustness of detectors against adversarial attacks.
GPT-Sentinel (Chen et al., 2023). Targeting the problem of machine-generated text detection, Chen et al. (2023) propose two simple approaches: (1) Training a simple lincar classifier on top of a pre-trained RoBERTa model (Liu et. al., 2019), and (2) Finetuning a T5 (Raffel el al., 2020) model. For this pur- pose, Chen et al. (2023) curate dataset namely OpenGPTtext by paraphrasing textual samples from the OpenWebText (Gokaslan et al., 2019) corpus using the GPT-3.5-turbo model. Specifically, OpenGPTtext con- sists of 29, 395 samples, where each textual sample corresponds to human-written text from OpenWebText corpus. They demonstrate that both fine-tuning and linear-probing technique on pre-trained models result in enhanced performance in identifying machine-generated text in comparison to baseline techniques like those presented by Tian (2023), OpenAI (2023), and Solaiman et al. (2019). The fine-tuning approach exhibits better accuracy text detection than linear-probing, i.e., simply training a linear classifier on top of the frozen pre-trained model.
LLMDet: A Large Language Models Detection Tool (Wu et al., 2023a). To detect machine- generated text, Wu et al. propose a text detection framework namely LLMDet. A previous study by Mitchell et al. (2023) has highlighted the usefulness of perplexity in detecting calculating perplexity requires white-box access to the original model parameters which may be unrealizable in practice. To leverage the usefulness of perplexity without assuming white-box access, Wu et al. introduce the notion of calculating a proxy score for perplexity. The text detection framework consists of two phases: (1) the Dictionary phase, and (2) the Training phase. In the dictionary phase, n-grams and their corresponding probabilitics are stored as keys and values respectively in a dictionary. First, the language model prompted repeatedly to generate a collection of text samples. Next, n-gram word frequency statistics are calculated the set of generated text. Finally, n-grams and their corresponding probabilities are stored as keys and
text. However,
16

---

Published in Transactions on Machine Learning Research (01/2024)
values respectively in dictionary. The probability for a n-gram is calculated based on frequency statistics of text generated by the language model. In the training phase, information stored in the dictionary phase is used to calculate the proxy perplexity. Finally, these proxy perplexities are used to train text classifier to distinguish between machine-generater text and human writing.
How Close is Chat.GPT to Human Experts (Guo et al., 2023)? The contributions of Guo el al. are essentially two-fold: (1) First, they aim to shed light on how close are state-of-art LLMs to human writers, Specifically, they provide an understanding of the properties of text generated by ChatGPT and how it differs from human writings. (2) Second, to minimize risks related to Al-generated content, they propose different detection systems. To understand the differences in text generated by ChatGPT and human writers, they curate a Human ChatGPT Comparison Corpus. In the corpus, each instance is a question followed by answers from a human expert and a response generated using ChatGPT. Using this dataset, they conduct two specific tests. In the first test, referred to as the Turing test, given a question and corresponding response (can be from a human writer or Chat a human evaluator has to identify whether the response is machine-generated or not. In the second setup, referred to as the Helpfulness lest, given a question and responses from a human writer and ChatGPT, the task is to evaluate which response among the two is more helpful. Based on this analysis, they observe: (1) Responses generated by ChatGPT are correctly identified by a human evaluator approximately 80% of the time. (2) ChatGPT-generatee answers are in general more concrete and helpful than human writers specially for questions from the domain of finance and psychology. However, for questions related to medical domain ChatGPT generated answers are not much helpful and poorly crafted as compared to human writings.
Further, for the purpose of detecting AT-generated text, Guo et al. (2023) consider: (1) Training a logistic regression classifier on the features obtained from GLTR (Gehrmann et al,, 2019) test, and (2) finetuning strong pre-trained transformer, specifically RoBERTa (Liu et al., 2019). They found that the RoBERTa- based detector is more robust and also exhibits stronger detection performance as compared to GLTR.
Ghostbuster: Detecting Text Ghostwritten by Large Language Models (Verma et al., 2023).
Several studies, as discussed in the previous subsection (Mitchell et al., 2023) have explored the detection of machine-generated text by analyzing token pog-prbbabilities. However a common challenge in generating the token probabilities involves assuming whitebox access to the target model. In their recent work, Verma et al. (2023) propose Ghostbuster, detection framework that does not require access to token probabilities from the targel model. This enables delection of texl generated from unknown models. To learn the token log probabilities, during training Ghostbuster passes each document through a series of less powerful language models. However, instead of performing classification directly based on generated log probabilitios, the token probabilities are passed through a series of vector and scalar operations. These operations combine the token probabilities giving rise to additional synthetic features. To illustrate, consider P1 and P2 be the generated probability vectors for any two tokens. Some instances of vector operations include addition (pI + P2), subtraction (P1 - P2), multiplication (P1 P2) and division (pz/pz). Scalar operations include computations like maximum (max p), minimum (min p), length (Pl) and 1-2 norm (I[p)12)- Apart from these synthetic features, Verma et al. proposed using a set of manually crafted features incorporating qualitative insights and heuristics observed in Al-generated text. Finally a logistic regression classifier is trained on top of these features to detect machine-genen generated text.
RADAR: Robust AI-Text Detection via Adversarial Learning (Hu et al., 2023). A signifi- cant challenge in identifying machine-generan text involves tackling paraphrasing attacks. Further, prior studies (Krishna et al., 2023; Sadasivan et al., 2023) have shown that (recursive) paraphrasing attacks can significantly reduce the detection performance. To mitigate this issue, Hu et al. (2023) proposed novel detection framework called RADAR. This framework employs an adversarial learning approach to simultanc- ously train a detector and a paraphraser. At a higher level, detector's objective is to accurately differentiate between authored content and machine-generasd volves generation of plausible and realistic text with the aim of cluding detection by the detector. Let Co. D&, represent the target language model, the detector and the paraphraser parameterized by B, Φ, and a respectively. Note that the target language model Lo is frozen through-out and does not involve any train-
text. In contrast, the paraphraser's 's training in-
17

---

Published in Transactions on Machine Learning Research (01/2024)
Target Language Model
Method
Dataset
Dolly-V2-3B
GPT-J-6B
Dolly-V1-6B LLaMA-7B
Vicuna-7B
Xsum
0.989
0.789
0.904
0.568
0.969
logp
SQuAD
0.983
0.754
0.909
0.599
0.910
WP
0.990
0.892
0.890
0.865
0.976
Xsum
0.894
0.543
0.465
0.701
0.398
entropy
SQuAD
0.502
0.566
0.501
0.613
0.394
WP
0.214
0.312
0.209
0.404
0.189
Xsun
0.985
0.768
0.820
0.678
0.883
SQuAD
0.954
0.700
0.614
0.552
0.723
DetectGPT
WP
0.911
0.803
0.841
0.726
0.960
Xsum
0.905
0.917
0.945
0.832
0.790
SQuAD
0.876
0.924
0.910
0.830
0.807
OpenAl
WP
0.875
0.904
0.909
0.875
0.870
Xsum
0.824
0.902
0.895
0.892
0.895
SQuAD
0.692
0.734
0.820
0.715
0.895
RADAR
WP
0.697
0.735
0.808
0.734
0.814
Table 1: Evaluating popular language models using state-of-art Post-hoc detectors on Xsum, SQuAD, and WP dataset. The table is motivated from Hu et al. (2023). The values are obtained by reproducing the results in (Hu et al., 2023).
ing. The detector D& and the paraphraser Go are initialized with pre-trained T5-large and RoBERTa-large models respectively. Let H. represent human-text. corpus, generated by sampling 160K documents from WebText (Gokaslan et al., 2019). Let M be a corpus of Al-generated text generated using the target lan- guage model Co, by performing text completion using the first 30 tokens as prompt. The training procedure involves two components:
Training the paraphrascr. First, Al-generated text samples Sm ~ M are fed to the paraphraser Go as input. Let be the output of the paraphraser and P be the corpus of all paraphrased samples. The paraphraser parameters are updated using Proximal Policy Optimization (Schulman et al., 2017) with the reward feedback from the detector Da. The reward returned by sp is the output of Do(sp), i.c., the predicted likelihood of 8p being human written.
Training the detector. As in a typical GAN training, the detector is tasked with accurately dis- tinguishing human-written samples from Al-generated and paraphrased counterpart. The detector is trained using logistic loss defined as: LD, LH + ALM + XLp, where Lx -EE, log(Da (sw) represents the loss to improve the correctness of predicting ~ H as human--witted LM E. log(1 - Ds(sm) and Lp = -Eg log(1 Do(sp)) represents the loss to predict Sm and Sp from boing predicted as human-text.
Empirical analysis highlights the strong text delection performance of RADAR on a suite of 8 targel LLMs. Specifically on XSun dataset, when samples are paraphrased using a OpenAl GPT-3.5- T-rbbo API (which is different from the paraphraser using during training RADAR), RADAR improves detection performance by 16.6% and 59.5% as compared a Roberta-based detector fine-tuned on WebText (Gokaslan al., 2019) and DetectGPT (Mitchell et al., 2023).
18

---

Published in Transactions on Machine Learning Research (01/2024)
5 Towards the Impossibilities of Al-generated Text Detection
5.1
Paraphrasing Based Attacks
5.1.1 Red Teaming Language Model Detectors with Language Models (Shi et al., 2023)
Attacks introduced in Sadasivan et al. (2023); Krishna et al. (2023) evade detectors by paraphrasing the Al-generated lext using a language model. However, for the proper functioning of the allack, they assume that the paraphrasing model is not protected by any detection mechanism. Shi et al. argues that this assumption is not always realizable in practice, as in the future all publicly available language models might have a detection framework in place as protection. To this end, Shi et al. (2023) proposes a more realistic attack mechanism for cases even when the paraphrasing model has a detection mechanism in place.
Given an input text prompt h, lct s = {s (sn) represent the output sequence of length N generated by language model Cg. Let Go represent another language model used for generating the altacks. Specifically, they consider two types of attacks:
The first kind of attack involves perturbing the output sequence s to generate a modified output s'. This is done by replacing certain tokens in s with substitute words maintaining the naturalness and original semantic meaning. For replacing a token Sş the original output s, Ge prompted to generate substitute candidates with the same meaning as 8g. Let & represent the set of substitute candidate tokens generated by Go. Given substitution budget e, the minimization framework is defined
s' arg min D(s')
s.t. s/ E {sk} U Sk.
N
i[s + st] s EN
where DD represents the text detector being attacked and T represents an indicator function.
The second kind of attack is based on perturbing the input prompt h to generate h', thereby leading to a modified output s'. The main idea is to modify the input prompt to shift the distribution of the text generated by the language, thereby evading the detector. This is done by appending an addit ional learnable prompt hp to the original input sequence h, leading to a new prompt h' [h, hp]- The additional prompt hp is searched by querying the detoctor multiple times using m input prompts ( J. The objective function of the search is defined as:
arg min 1 - I[P(Lo((h; hzl)) 2 6) hp
1=1
In simple words, the search function aims to find an additional prompt h, such that the average detection rate is minimized for the new outputs generated. This form of attack is primarily targeted for detection frameworks based on fine-tuning or training a neural classifier.
Empirical analysis by Shi al. (2023) shows that an attack based on modifying the output sequence causes an 88.6% degradation in AUROC for DetectGPT-based detector, 41.8% improvement as compared to paraphrasing based attacks. Here the attack is based GPT-2-XL model and XSum dataset.
5.1.2 Paraphrasing Evades Detectors of Al-generated Text (Krishna et al., 2023).
The paper questions the robustness of current statc-of-art text detection algorithms (Kirchenbauer et al., 2023a; Mitchell al., 2023) against paraphrasing attacks curated evade the detector. The authors argue that the current paraphrasing algorithms do not properly emulate real-world conditions and suffer from two
19

---

Published in Transactions on Machine Learning Research (01/2024)
fundamental limitations: (1) They are trained on sentences, ignoring paragraph-level information. (2) They lack knob to control output diversity. Hence, they might not be the ideal candidate to stress-test these text detection algorithms. Specifically as text is paraphrased more, it necessarily drifts away in meaning from the original text, making it less useful for an attacker. Previous paraphrasing attacks are trado-offs on this curve of textual similarity and paraphrasing strength. In this work, Krishna et al. (2023) use an explicit diversity parameter to control this trade-off. Improving on these limitations, Krishna et al. (2023) train an external paraphrase generator called DIPPER. The model is loaded from a T5-XXL (Raffel et al., 2020) checkpoint and is finetuned on the PAR-3 dataset (Thai et al., 2022) for paraphrase generation. Analysis by Krishna et al. (2023) shows that for the GPT2-XT, model, strongest paraphrasing attacks using DIPPER can degrade the detection capabilities of natermarkedd-asse (Kirchenbauer et al., 2023a) model and DetectGPT by 52.8% and 65.7% respectively.
5.1.3 Paraphrasing by Humans
In a related sctup, likely to occur in practice, the "attacker" might just themselves be a human writer who is paraphrasing a watermarked text with the purpose of evading detection. Interestingly, analysis in Kirchenbauer et al. (2023b) shows that human writers are indeed strong paraphrasers and are more capable of fooling detector as compared to machine-based paraphrasers, which in turn means that text paraphrased by human writers needs to be significantly longer, for the watermark of Kirchenbauer et al. (2023b) to be detected.
5.2 Generative Attacks
Generative attacks, or cipher attacks, were proposed in Goodside (2023) and are studied in Kirchenbauer et al. (2023a). These attacks exploit the capacity of large language models for in-context learning, prompting these models to modify their responses in a manner that is both foresceable and readily reversible. The essence of these attacks lies in subtly manipulating the model's output in a controlled manner, such that the attacker can predict the outcome and also revert the model back to its original state or responses if nceded. An interesting example of such carefully crafted generative attack is the emoji attacks shown by Goodside (2023). The attack strategy works by instructing the model to produce an emoji following each token it generates. The tricky part is that removing these emojis would randomize the red list for the subsequent tokens, thereby effectively evading detection by the watermark detector. Other examples of generative attacks studied in Kirchenbauer et al. (2023a) include prompting the model to replace all 'a' with 'el. Even other examples are prompts that ask the language model to return outputs completely different from the inputs, for example a language like Chinese or in a basc64 encoding.
These types of attacks effectively work by prompting to model to use cipher during generation, that completely changes the n-grams observed in the text, but is easily revertible by the attacker. Compared to other attacks, such as edits or paraphrases, which only modify some of the tokens in the response, and thereby dilute a signal present in the texl, these attacks can change the generated lext completely, thereby fully removing, e.g. the watermark of Kirchenbauer et al. (2023a). However, executing these attacks requires a language model capable enough to adhere to the instructed rule without overly compromising the quality of its output. Kirchenbauer et al. suggests that one viable defense strategy against these attacks could involve incorporating adverse examples of such instructions during the fine-tuning process, thereby training the model to decline such requests.
5.3 Copy-Paste Attacks
Introduced in Kirchenbauer et al. (2023b), this is a form of synthetic evaluation where Al-generated text passages are embedded within a longer human-wrilten document. This results in sub-spans of text sequences having an abnormally high number of tokens from the green list. Kirchenbauer et al. modulates the attack strength using two parameters: (1) the number of Al-generated sequences inserted in the document, and (2) the fraction of the documentt that represents Al-generated text. The empirical analysis in Kirchenbauer al. (2023b) shows that usual detection approaches struggle when faced with text that is diluted in this manner,
20

---

Published in Transactions on Machine Learning Research (01/2024)
and need to be adapted to include windowing scheme to still detect the smaller sequences of Al-generated text embedded in the document.
5.4 Spoofing Attacks
When an adversarial human deliberately generates a text passage to be detected as AI-generated, it is referred to as a spoofing atlack (Sadasivan et al., 2023). The goal of these atlacks could be to trick a detector into falsely classifying derogatory human writings as Al-generated with the malicious intention of harming the reputation of an LLM developer. Sadasivan et al. also provides approaches for spoofing various detection mechanisms. Specifically for watermarked models (Kirchenbauer et al., 2023a), the allack involves computing a proxy of green list for some of the most commonly used words in the vocabulary. For watermark with a context width of 1, the attack in Sadasivan et al. queries the watermarked model large number of times (for OPT-1.3B model, 106 queries are computed) and observes pair-wise token occurrences in the output to generate an independent estimate the green list for each context (or just each token for a context width of 1). Then, after learning the proxy for these green lists, an adversarial human can generate text to be misclassified as watermarked by inserting these token sequences.
However, the hardness of this attack is directly related to the context width of the watermark, and attacks watermarks with longer context widths, c.g. width of 4 as in Kirchenbauer et al. (2023b) have far not been demonstrated. For the watermark of Christ et al. (2023a), this type of attack has been proven to be intractable.
For rotrioval-based detection (Krishna et al., 2023), Sadasivan et al. proposes to give human-written passages to the paraphraser. Recall that in the detection scheme proposed by Krishna al. (2023), the output of the paraphraser model stored in a database. Thus, all the paraphrased human writings are also fed to the database. Now, during inference, human-written passage would achieve a high similarity score with the paraphrased human writings in the database, thereby fooling the detector to misclassify the human-written text as AI- Al-generated.
Sadasivan et al. (2023) also demonstrates successful spoofing attacks for zero-shot and trained detectors (like OpenAl's Roberta-bass detector) where they first prepend the original human-written text with other texts that were poorly detected by the detectors. The detection performance drops drastically (drastic increase in FPR at a fixed TPR) by spoofing attack demonstrating the efficacy of the spoofing attack. Thus the interestingly designed spoofing attack is shown to be successful against majority of the detection and watermarking schemes (Sadasivan et al., 2023) and needs careful consideration. Although successful, such attacks raise a natural question of the motivation and purpose of adversarial and spoofing humans and applications where such attacks are practical like social media spoofing, online chat spoofing, etc. and we discuss the scenarios in Section 6.6.
6 Open Questions
In this section, we emphasize several critical questions and directions pertaining to the research on AI- generated text detection which are still unsolved and need further discussion.
6.1 Regulating Large Language Model Watermarks.
There is an ongoing world-wide debate on whether some form of regulation should be imposed on large lan- guage models to prohibit their use for nefarious purposes such as spreading disinformation impersonating individuals. A plausible solution could be to set up regulatory body to watermark all publicly available language models and install detection mechanism as protection. A potential variant of this approach is that all watermarked models share a universally standardized detection key. Otherwise, it is possible to cir- cumvent the watermark detector by diluting the distribution of tokens via paraphrasing with a second model with a different watermark. For example, let s represent an output sequence generated by a language model Co watermarked with key k. Let Go be paraphrasing model, watermarked with a different detection key W. Then although watermarked, paraphrasing by G6 can still modify the distribution of tokens being
21

---

Published in Transactions on Machine Learning Research (01/2024)
sampled from the green list, thereby evading detection of the first model, and only partially encoding the watermark of the paraphrasing model. On the other hand, regulation could also adopt principle of "last model responsibility", where responsibility for Al-generated text falls onto the company whose watermark is detected, even if the text may be originally sourced from another model.
The potential of regulation further highly related to the amount of democratization in the ecosystem of large language model. If only few companies dominate and produce most of the Al-generated text, then coordination is casy. If text generation is split across many independent actors, then attackers might always prefer, directly control, those models that are not watermarked.
From a broader perspective though, regulation of large companies may be strong approach to reduce the overall impact of Al-generated lext, even if it is only making detection easy in all non-adversarial scenarios, Lee documentation Al-generated text, accidental mis-attribution, etc. Smaller open-source models and smaller companics could be exempt until a significantly large fraction of AI-gencrated text is made by their models.
6.2 Accurate Evaluation of Detectors
In the past few years, the natural language processing community has witnessed deluge of studies being published trying to solve the problem of Al-generated text detection. The majority of these studies, promote their proposed framework by showing an improved accuracy in distinguishing machine-generated text from human writings, In most of these works, generally, texts from datasets such as XSum (Narayan et al., 2018) and SQuAD (Rajpurkar et al., 2016) are used as a proxy to characterize the human distribution. However, there may be a huge diversity between the original and modeled human distribution. For example, from the context of text generation, the fluency of answers written by linguistic experts or native speakers will be very different from the ones authored by non-native speakers or writers. Further, the recent observation by Liang al. (2023) that perplexity- based detectors are biased against non-native speakers highlights the urgency of an accurate characterization of human distribution to train these detectors. This naturally leads to challenging question: how can we accurately evaluate detectors on broader parts of the human language distribution?
6.3 Conditions for Al-generated text Detection
The origin of debate regarding the possibilities and impossibilities of Al-generated text detection originated from Sadasivan et al. (2023). Specifically, Sadasivan et al. (2023) posit theoretical arguments stating that when there is a strong overlap between machine and human-generatee to detect. AT-generated text. However, recent. work by Chakraborty et al. (2023) derive precise sample complexity analysis and show that it is almost always possible to detect Al-generated text from information- theoretic principles if one can collect enough samples or increase the sequence length. Corroborating the analysis presented by Chakraborty et al. (2023), Kirchenbauer et al. (2023a) also recently show that increasing the number of tokens significantly plummets the success of a paraphrasing attack. This lcads to a natural question: in which settings one can always obtain additional samples? the case of detecting text generated by interactive Chatbots or Twitter bots, one can always prompt the model multiple times obtain enough training samples for obtaining perfect detection accuracy. However, it may not always be the case. Consider educational sctup, where the task is to check for the originality of a student-authored essay. In the worst case, the distribution of the essays written by all students in class may closely follow the distribution of machine-generated essays, Since, in this scenario, collecting additional samples is impossible, according to the analysis presented by Sadasivan et al. (2023), one can only do as good as a random detector. This leads to another open question: can we improve detection performance without assuming the availability of additional samples?
text distributions, is impossible
6.4 Improving the Fairness of Text Detectors.
Although there has been a deluge of studies in the last few years on improving machine-generated text detection, recent study by Liang et al. (2023) highlights serious concerns regarding implicit biases present in
22

---

Published in Transactions on Machine Learning Research (01/2024)
GPT-based detectors against non-native English writers. Specifically, Liang et al. (2023) observe perplexity- based detectors having high misclassification rate for non-native authored TOFFL, essays despite being nearly perfectly accurate for college essays authored by native speakers. Liang et al. hypothesize that a plausible explanation for this biased behavior could be because of low perplexity in text written by non-native English speakers due lo lack of linguistic variability and richness. Biases lext delectors can have serious repercussions educational settings such as falsely accusing a marginalized group of plagiarism. Thus there is a dire need to focus on mitigating bias and improving fairness in text detectors based on perplexity scores or classifiers.
This problem is especially pronounced in black-box detection system based on finetuned classificrs, which depend strongly on the type of text they are trained on, but can also appear in zero-shot detectors if they are not carefully evaluated and tuned. On the flip-side, watermark detection is unaffectdd, as it is based on a null hypothesis that holds independently of the input text. Retrieval likewise also unaffected, the retrioval model is unbiased.
6.5 Improving the Robustness of Text Detectors.
Based on the comprchensive discussion of various attack frameworks, we make the following key observations: (1) It is clearly evident that recursive application of paraphrasing atlacks can exert an adverse impact on the efficacy state-of-art text detectors (Sadasivan et al., 2023; Krishna et al., 2023). It is also important to note that while employing an iterative randomized paraphrascr is guaranteed to eventually remove all traces of the original text including watermarks, it invariably introduces alterations in the original text's meaning, So, there is always a trade-off between the strength of the paraphrasing attack and the extent to which the text's meaning shifted. (2) On the other hand, generative attacks exhibit a bijective nature and can even evade the detector without causing a substantial shift in text quality and meaning.
In light of these insights, how attacks still remains an open and critical challenge. Additionally, it is also imperative to acknowledge that the feasibility of the detection is inberently contingent on the objectives and constraints of the stakeholders involved. To illustrate, within an educational context, instructors might want to catch students who wrote essays using machino-gonerated text. In such scenarios, detection can be made casier if the instructor can beforehand generale characterization of the distribution of maahine--gneeatter essays by querying the language model multiple times. There can be other scenarios, including: (1) Individuals aiming accumulate evidence to ban LLM actors from social platforms such as Twitter, and (2) Those wanting to document language model usage to prevent dilution of the internet with machine- machine-generater text. So, to summarize, the leasibility of detection is intrinsically linked to the specific requirements of the user such the amount of text that needs to be detected, acceptable false-positive rates, and acceptable threshold of evidence.
design more robust detector watermarking scheme against potential
6.6 Discussion on Different Detection Scenarios in Practice
The discourse on Al-generatod text has been the focal point of extensive discussions recently, but the existing literature often lacks a comprehensive exploration of the diverse practical scenarios necessitating detection, their specific requirements, and the importance of accurate detection outcomes. Specifically, we note that it is imperative discern the distinct demands depending upon the context, urgency, and gravity of detection in order to effectively understand both the potential and limitations of Al-generated text detection. In this section, we aim to shed light on a range of scenarios where the detection of Al-generated text is of utmost importance and connect it to the requirements of detection and available possible solutions in the literature.
Fake News, Misinformation, and Social Media Content Moderation
Concern: The potential for bias and misinformation spread through Al-generated content on social modia platforms and reviews, typically via bots (De Angelis et al., 2023).
Scenarios:
Social Media: Need to detect Al-generated spam or malicious content for safe online environment.
23

---

Published in Transactions on Machine Learning Research (01/2024)
Detecting fake reviews generated by bots AI to ensure the credibility of product and service reviews.
User Reviews:
Information Verification: The assossment of the authenticity of articles and posts is crucial in combating the dissemination of misinformation. Past studies (Pegoraro et al., 2023; Day, 2023; Gravel et al., 2023) have shown that language models can be employed to craft misleading fake articles there by necessitating robust verification techniques.
Deepfakes: Detection of AT-generated text used in conjunction with deepfake videos or im- ages is essential to counter disinformation campaigns. Deepfake technologies, when coupled with Al-generated text, can create compelling false narratives, posing a significant threat to public discourse and trust.
Note: In the above scenarios, gathering multiple samples at least from the social bots is feasible. The samples may be collected from the same social bots or group of identical bots. Based on the theoretical and empirical analysis in Chakraborty et al. (2023); Kirchenbauer et al. (2023b), the collected i.i.d samples can then be used further to enhance the reliability of Al-generated text detection methods.
2. Academic Integrity
Concern:
The potential misuse of AT in creating essays or assisting students dishonestly.
Scenarios:
Plagiarism Detection:
Detecting Al-generated essays or reports submitted as original work
by students in educational institutions is persistent concern (Sullivan et al., 2023; Currie, 2023; Perkins, 2023).
Cheating Prevention: Identifying automated tools used by students to cheat in online exams or assignments (Cotton et al., 2023).
Nole: In some specific scenarios such as in education systems, wrong detection can lead to false penalization of students, thereby causing drastic repercussions on their careers. Hence, for such scenarios, it becomes crucial to minimize false positives while detecting Al-generated content. However, we note that interestingly, in these scenarios, since the prompts are always available to the instructor, it is possible to learn the machine distribution by querying open- source models. Thus, detection becomes easier and boils down to identifying distr-iiist samples in that context. Additionally, the instructor can ask for minimum requirement of sentence length to enhance detection (Chakraborty et al., 2023). Thus, although the problem of detection challenging, there are alternatives to deal with the same, which is an interesting direction for future research.
3. Financial Services
The use of AT in generating phishing messages, especially when mimicking financial bodies (Bateman, 2020).
Concern:
Scenarios:
Fraud Detection: The identification of Al-generated phishing emails or messages used in fraud attempts, especially when impersonating banks or financial institutions, is a critical application (Bateman, 2020).
Identity Verification: Accurately determining whether text responses in identity verification processes are generated by humans or automated systems is vital for frand prevention.
Note: To minimize the threats, a possible solution can be: (1) watermark all publicly avail- able language models, and (2) design email filtering schemes which can detect the embedded watermark in the phishing messages and flag them for further verification.
4. Customer Support
Concern:
Differentiating between Al and human -ennan-geeeraedd responses.
Scenarios:
24

---

Published in Transactions on Machine Learning Research (01/2024)
Chatbots: In distinguishing between responses generated by AI chatbots and those from human customer support agents, transparency and quality assurance are key concerns (Jain et al., 2019). While it may be relatively easier to identify Al-generated text in an online interaction with AI agents, the potential presence of adversarial humans requires vigilance. The presence of adversarial humans this scenario highlights the potential of a spooling attack where adversarial humans can manipulate the chat to fool the detector into believing that it is a chatbot.
Email Responses: Identifying automated responses in email communication from businesses or customer service departments is essential to maintain customer trust and ensure efficient
communication.
Note: In the above scenarios, certain strategies, such as asking specific questions that Al may struggle to answer naturally, aid detection.
5. Intellectual Property
Concern:
The unauthorized use of Al to reproduce copyrighted materials.
Scenarios:
Copyright Violation: Detecting Al-generated text used in the unauthorized reproduction of copyrighted content, such as books or articles, is vital to protect intellectual property rights (Zhong et al., 2023). Automated tools can generate content that infringes copy- rights, necessitating robust detection mechanisms (Wu et al., 2023b).
Note: A plausible solution would be to embed watermarks (Kirchenbauer al,, 2023a) into digital content, to make it easier to identify the source and provenance of the content.
6.7 Can We Leverage Our Knowledge of Input Prompts for Better Detection?
In this section, we look into a compelling question: does foreknowledge of the input prompt (to be used by an adversary), simplify the detection task? We contemplate that under certain circumstances, having advanced insight into the input prompt could prove advantageous. To illustrate, let's consider an educational context: during an exam, students are asked to compose an essay on "computers". Also, let us assume that the question is framed as: "Compose an essay on computers". In this setting, it would also be fair to assume, that students who are planning to resort to unfair means such as using a language model, would use the original question or a slightly paraphrased version as the input prompt. Then, for the purpose of detecting plagiarized essays, the instructor can beforehand compute a characterization of the distribution of machine- generated essays by querying the language model multiple times using the same question prompt: "Compose essay on computers".. Consequently, any student employing language models for writing the essay can be detected by calculating some form of similarity with the pre-established machine distribution. We believe studying how this approach can be expanded to other scenarios represents a promising avenue for future rescarch.
6.8 Is Semantic Watermarking Possible?
All watermarking strategics discussed in this work depend on the form a particular piece of text, not on content. In this way, these watermarks are easy to deploy, as they are designed not to modily the content of the text, but are also fundamentally vulnerable to modifications of the text that change its form, but leave its semantic content unchanged, such as edits, paraphrases and generative attacks. This problem seems only solvable with watermarks that operate not the form of the text, but higher-level semantics. How to design watermarks that operate on this level, while still including beneficial properties of non-semantic watermarks, such as undetectability, analytically defined p-values and low false positive rates, is so far an open question.
7 Conclusions
In this survey, we review works highlighting the possibilitics and impossibilities of detecting machine- generated text. Specifically, provide a precise categorization and in-depth analysis of various detection
25

---

Published in Transactions on Machine Learning Research (01/2024)
frameworks and attacking schemes. To further benefit the ongoing research, we discuss challenging open research questions that still need addressing. We hope that our survey will be beneficial to the community in understanding the strengths and limitations of existing research related to Al-generated text detection.
References
Scott Aaronson. 'reform' ai aligmment with scott aaronson, 2023. URL attps://axrr.nntteepiisss 04/11/episode wwweoodde----rforro ai -aligamentsscooeeeeeeeeee accott-aroonoo html.
Mikhail J Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mo- hamed, and Sanket, Naik. Natural language watermarking: Design, analysis, and proof-of-concepl im- plementation. In Information Hiding: 4th International Workshop, IH 2001 Pittsburgh, PA, USA, April 25-27, 2001 Proceedings 4, pp. 185-200. Springer, 2001.
Mikhail Atallah, Victor Raskin, Christian F Hempelmann, Mercan Karahan, Radu Sion, Umut Topkara, and Katrina E Triezenberg. Natural language watermarking and tamperproofing. In International work- shop information hiding, pp. 196-212. Springer, 2002.
Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-11, 2008.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906. 03351, 2019.
Jon Bateman. Deeyfakes and synthetic media in the fonancial system: Assessing threal. scenarios. Carnegie Endowment for International Peace., 2020.
Daria Beresneva. -omputerr-eenerated text detection using machine learning: A systematic review. In Natural Language Processing and Information Systems: 21st International Conference Applications of Natural Language to Information Systems, NLDB 2016, Salford, UK, June 22-24, 2016, Proceedings 21, pp. 421-426 Springer, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krucger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Nicholas Carlini, Matthew Jagiclski, Christopher A Choquetto-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramer. Poisoning web-scale training datasets is practical. arXiv preprint wwXXi:2302.10144 2023.
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. aix preprint arXim:23040447 2023.
Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Ramakrishnan. Gpt-sentinel: Distinguishing human and chatgpt generated content. arXiv preprint arXio:2305.0000 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gchrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani
26

---

Published in Transactions on Machine Learning Research (01/2024)
Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polawov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. Cryptology ePrint Archive, 2023a.
Miranda Christ, Sam Gumn, and Or Zamir. Undetectable Watermarks for Language Models, 2023b. URL https://eprint. iacr. iacr. org/2023/763.
Debby RE Cotton, Peter A Cotton, and J Reuben Shipway. Chatting and cheating; Ensuring academic integrity in the era of chatgpt. Innovations in Education and Teaching International, pp. 1-12, 2023.
Geoffrey M Curric. Academic integrity and artificial intelligence: is chatgpt hype, hero or heresy? In
Seminars in Nuclear Medicine. Elsevier, 2023.
Falcon Dai and Zheng Cai. Towards near-imperceptible steganographic text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4303-4308, 2019.
Terence Day. A preliminary investigation of fake peer-reviewed citations and references generated by chatgpt. The Professional Geogropher, pp. 1-4, 2023.
Luigi De Angelis, Francesco Baglivo, Guglicimo Arzilli, Gactano Pierpaolo Privitera, Paolo Ferragina, Al- berto Eugenio Tozzi, and Caterina Rizzo. Chatgpt and the rise of large language models: the new ai-driven infodemic threat in public health. Frontiers in Public Health, 11:1166120, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ar arxi::1800.00 2018.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. Tweepfake:
About detecting deepfake tweets. Plos one, 16(5)=x0251415, 2021.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedingss of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889 898, 2018.
Pierre Fernandez, Antoine Chaffin, Karim Tit,, Vivien Chappelier, and Teddy Furon. Three bricks con- solidate watermarks for large language models. arXiv preprint arXiv:2308. N.f:2338800013,,, 2023.
Markus Freitag and Yaser Al-Onaizan. Beam scarch strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pp. 56-60, Vancouver, August 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1/W17--
URL https::/ http:///iaanataooooooooo///11110 org/M17-32077.
Margherita Gambini, Tiziano Fagni, Fabrizio Falchi, and Maurizio Tesconi. On pushing deepfake tweet detection capabilities to the limits. In Proceedings of the 14th ACM Web Science Conference 2022, pp. 154-163, 2022.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arxi::1906 04043, 2019.
Aaron Gokaslan, Vanya Cohen, Fllie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019. URL http: //Skylion007 github. httpb//oo/oeeeeeeettooooooo
Riley Goodside. There are adversarial attacks for that proposal as well in particular, generating with emojis after words and then removing them before submitting defeats it, 2023. URL https://twitter.
Jocelyn Gravel, Madeleine D' /Amours-Gravcl and Esli Osmanlliu. Learning to fake it: limited responses and fabricated references provided by chatgpt for medical questions. Mayo Clinic Proceedings: Digital Health, 1 (3):226 234, 2023.
27

---

Published in Transactions on Machine Learning Research (01/2024)
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. orNiv preprinl 0rXiv:2301007500 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-300 3938. PMLR, 2020.
Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting intellectual prop- erty of language generation apis with lexical watermark. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 36, pp. 10758 10766, 2022a.
Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. Cater: Intellectual property protection on text generation apis via conditional watermarks. Advances in Neural Information Processing Systems, 35:5431-5445 2022b.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yojin Choi. The curious case of neural text degener- ation. In International Conference on Learning Representations, 2019.
Dirk Hovy. The enemy in your camp: How well can we detect statisticalll-generate fake reviews- adversarial study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Popers), pp. 351-356, 2016.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint ar xrx:::1801.000
2018.
Xiaomeng HU, Pin-Yu Yu Chen, and Tsung-Yi Ho. Radar: Robust ai-text detection via adversarial learning. arXiv preprint XrXi::2307.038 03838, 2023.
Matthias Huck, Dario Stojanovski, Viktor Hangya, and Alexander Fraser. Lam munich's neural machine translation systems at wmnt 2018. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 648 648-654, 2018.
Shreya Jain, Dipankar Niranjan, Hemank Lamba, Neil Shah, and Ponnurangam Kumaraguru. Characterizing and detecting livestreaming chatbots. In Proceedings of the 2019 IERE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp. 683-690, 2019.
Ganesh Jawahar, Muhammad Abdul-Magerd and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arX:::2011.0111, 2020.
Stefan Katzenbeisscr and Fabicn Petitcolas. Information hiding. Artech house, 2016.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXix:2301. 10226, 2023a.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblun, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.0 04634, 2023b.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint Xiw::2903.13408, 2023.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion frreee water- marks for language models. arXiv preprint arXiw:2307.1555 15508, 2023a.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust Distortion-free Wa- termarks for Language Models. arziv:2307.15598fey July 2023b. doi: http://arxiv org/ abs//bb2/7 15593.
URL
Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. Pan, 8(27331):4, 2008.
28

---

Published in Transactions on Machine Learning Research (01/2024)
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. GPT detectors are biased against non-native english writers. In ICLR 2023 Workshop on Trushworthy and Reliable Large-Scale Machine Learning Models, 2023. URL
Aiwei Liu, Leyi Pan, Xuming Hu, Shu'ang Li, Lijie Wen, Trwin King, and Philip S Yu. A private watermark for large language models. arXiv preprint or orXiu:3307.16230 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprinl ar Xiv: 1692, 2019.
Fatemehssdaa Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick, Smaller language models are better black-box saahine--eneratee text detectors. arXiv preprint or orXiv:2305.09859 2023.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine--gnneaaeed text detection using probability curvature. arv preprint ar 2xXx:23301113000 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745, 2018.
OpenAL. Chatgpt:
Optimizing language models for dialogue. 2022.
URL https://opena..co//a
chatgpt/-
OpenAL.
New ai classifier for indicating ai-written text. 2023. URL https://opemaa.com/bbll
Alessandro Pegoraro, Kavita Kumari, Hossein Fercidooni, and Ahmad-Reza Sadeghi. To chatgpt, or not to chatgpt: That is the question! arXiv preprint Sit::2344014877 2023.
Mike Perkins. Academic integrity considerations of ai large language models in the post-pandemic era: Chatgpt and beyond. Journal of University Teaching & Learning Practice, 20(2):07, 2023.
Jipeng Qiang, Shiyu Zhu, Yun Li, Yi Zhu, Yunhao Yuan, and Xindong Wu. Natural language watermarking via paraphraser-based lexical substitution. Artificial Intelligence, 317:103859, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin RafTel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1)):558555551 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Melhods in Natural Language Processing, pp. 2383 2392. Association for Computational Linguistics, 2016.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint Xrx::22303. 11156, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: (tt::1777.000 003440 2017.
Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsich. Red teaming language model detectors with language models. arXie preprint 2tl::23055.97113 2023.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arxi 190.00000, 2019.
29

---

Published in Transactions on Machine Learning Research (01/2024)
Chris Stokel-Walker. Ai bot chatgpt writes smart essays-should academics worry? Nature, 2022.
Miriam Sullivan, Andrew Kelly, and Paul McLaughlan. Chatgpt in higher education: Considcrations for academic integrity and student learning. 2023.
Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXie:2102.0955 2021.
Katherine
Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit Tyyer. Exploring document-level literary machine translation with parallel paragraphs from world litera- ture. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
9882-9900 2022.
E Tian. Gpizero. 2023. URL https://ggtzerr..... .me/.
Umut Topkara, Mercan Topkara, and Mikhail J Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In Proceedings of the 8th workshop on Multimedia and security, pp. 164-174, 2006.
Honai Ucoka, Yugo Murawaki, and Sadao Kurohashi. Frustratingly easy edit-based linguistic steganography with masked language model. In Proceedings of the 2021 Conference of the North American Chapter of the Associalion for Compulational Linguislics: Human Language Technologies, pp. 5486-544 2021.
Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. Decoding with large-scale neural language models improves translation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1387-1392, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrillen by large language models. arXie preprint arXiv:2305.1500 2023.
Max Wolff and Stuart Wolff Attacking neural text detectors. arXiv preprint orXiv:2002. 2020.
Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Limdet: A large language models detection tool. arXiv preprint arxiv:2905. 15004, 2023a.
Xiaodong Wu, Ran Duan, and Jianbing Ni. Unveiling security, privacy, and ethical concerns of chatgpt. arXiv preprint ar 231:22307.4419, 2023b.
Wenhan Xiong, Xiang Lorraine Li, Srini lyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kicla, et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint Xxxx:2009.12750 2020.
Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305. 17359, 2023.
Rowan Zellers, Ari Holtaman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Hanlin Zhang, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks the sand: Impossibility of strong watermarking for generative models, 2023.
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter J. Liu. gap-sentences for abstractive summarization, 2020.
Pegasus: Pre-training with extracted
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu- Yu-Xiang Wang. Provable robust watermarking for ai- generated text. Xiv preprint arXiv:2306.1 17439, 2023a.
30

---

Published in Transactions on Machine Learning Research (01/2024)
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible water- marking. arXio preprint or 031:2300.03162, 2023b.
Haonan Zhong, Jiamin Chang, Ziyue Yang, Tingmin Wu, Pathum Chamikara Mahawaga Arachchige, Chehara Pathmabandu, and Minhui Xue. Copyright protection and accountability of generative ai: At- tack, watermarking and attribution. In Companion Proceedings of the ACM Web Conference 2023, pp. 94 98, 2023.
Zachary Ziegler, Yuntian Dong, and Alexander M Rush. Neural linguistic steganography. In Proceedings of the 2019 Conference on Empirical Methods in Nalural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1210-1215, 2019.
31


Reviews:
[{"review_id": "G6u90TmoLZ", "reviewer": "Reviewer_ZHjB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does a good survey of a new but important topic. Overall speaking the strength outweighs the weakness. The clarity in writing, both high-level taxonomy design and some details can be improved.\n\nStrengths. The paper:\n\n- Addresses a timely and important topic to the research community.\n- Provides both theoretical and practical details of detection and anti-detection.\n- Summarize and proposed a list of research questions\n\nWeakness:\n\n- Taxonomy design. The paper organizes all algorithms into detection/anti-detection which further split into different methodologies (paraphrase / watermark etc). However Reviewer can see that it's common for a paper/algorithm to be both in the detection/anti-detection category by focusing on a particular aspect, e.g. paraphrasing. When reading the paper, Reviewer got a feeling that techniques like paraphrasing get mentioned repetitively. Another example of \"hidden\" dimension is whitebox vs blackbox detection.\n\n- Clear comparison of performance. The quality of detection is usually described as just \"strong\" without quantitative comparison to others. Reviewer understands that there are no consistent benchmarks and individual papers may not provide a readily available number. However it's still useful if the paper can give a tabulated overview with as detailed metrics as possible. Reviewer does notice that a better evaluation framework is one of the research questions proposed.\n\n- Minor unclarity in writing. See the requested changes section for the list.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a review about the topic of detecting text generated by large language models. This is a very recent field with significant social impact. The paper organizes related works into two categories: possibility (how to detect) and impossibility (how to avoid detection). It starts with a summary of theoretical discussion on if such a detection is possible or not. There are arguments in both camps. Then the paper discussed algorithms. The detection algorithms split into watermarking techniques which need full access to the model internals, and blackbox detection that doesn't. The anti-detection algorithms use various paraphrasing techniques to remove the watermark or signature of distribution from the model generated text.\n\nAfter introducing related works, the paper proposed a number of open questions for the research community to consider.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add performance numbers to each algorithm, even just as reported, instead of saying \"strong\". Preferably in a table.\n\n- Section 1.1. Be consistent about wording: \"preemptive\" vs \"prepared\"\n\n- Section 1.1, \"Towards the Impossibilities of AI-GTD\", \"three\" major categories should be \"four\"?\n\n- Figure 2. We should add world knowledge or context to the picture.\n\n- Section 3, first paragraph, \"text detectors [based on] [using] watermarking ...\" repetitive.\n\n- Section 4.1.2, second paragraph, \"...first to propose [a a] watermark..\" typo?\n\n- Page 10 4th paragraph, \"... blackbox access to the language model [G]iven an adversary...\" should be lower case?\n\n- Page 11, discussion about (Qiang et al 2023)'s method. The description of the algorithm is hard to understand. E.g. \"s_i == Para(y_i)\", does it mean one of the paraphrasing candidate of y_i matches, or top? And why do we need to sort list C in alphabetical order? Is there any pre/post processing to ensure fluency after token level substitution?\n\n- Page 14 \"Bot or Human...\" This is more about bot detection than text detection. Reviewer feels that it's either should not be in this survey, or deserves its own category.\n\n- Page 16. \"GPT-Sentinel ...\" last sentence, what is \"linear-probing method\"?\n\n- Section 5.2 first paragraph, \"... in a manner that is both [forseeable and readily reversible] ...\" not clear what it means when first reading about it. Suggest some clarification right away.\n\n- (minor suggestion) Page 20 last paragraph, \"... Sadasivan et al. proposes to [input] human-written...\" input -> give?\n\n- Section 6.5 first paragraph, \"(2) on other hand\" -> \"on the other hand\"? \n\n- Section 6.6 (1), Note section: \"atleast\" -> \"at least\"\n\n- Section 6.6 (4), \"potential presence of adversarial human\" could you give an example?\n\n- Section 6.7. Since the use of prompt/foreknowledge has been discussed multiple times in the paper, to this point the question feels quite repetitive to the Reviewer (although it is a valid research question). This could be another example of having better design of dimension in the taxonomy.", "broader_impact_concerns": "There are no concerns about the broader impact of the work. This is an important topic to make sure LLM-era of AIs can be used in a positive way for the society.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6S0xrBl7k5", "reviewer": "Reviewer_ZsKH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks to the authors for the hard work on this paper. I am not an expert in this sub-topic, so my comments are unfortunately limited.\n\nStrengths\n------------\n- A thorough discussion of many recent papers\n- Clear writing\n- Good organization\n- Nice outline of open questions\n\nWeaknesses\n----------------\n- I recommend removing the word \"Towards\" from the title. I am not sure what it means to be going towards a survey. You are already there.\n\nTypos\n-------\n- Page 4: \"two major groups based on usability: preemptive\" - do you mean \"prepared\"?\n- Page 10: \"with only black-box access to a language model Given an adversary\" - missing period before \"Given\"?\n- Page 10, bottom: \"Kirchenbauer et al.\" has no year, while all other references do\n- Page 13: \"4.2.1 Zero-shotDetection\" - missingspace\n- Page 19: \"an attack on based on modifying\" - extra \"on\"\n- Page 23: \"6.6 Discuss About the Different Detection Scenarios\" - should be \"Discussion on Different Detection...\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A thorough literature review about recent work in AI-generated text detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would be nice to change to the title, but it's optional...", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yopINGHOUM", "reviewer": "Reviewer_akKj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**:\n- The paper provides a comprehensive taxonomy and analysis of recent work on AI-generated text detection. Categorizing methods into \"towards possibilities\" and \"towards impossibilities\" offers a useful framework.  \n\n- Thoroughly reviewing detection schemes like watermarking, zero-shot, and classifier-based methods gives a good overview of current techniques. Discussing attack schemes like paraphrasing, generative, copy-paste, and spoofing attacks highlights vulnerabilities.\n\n- Connecting theoretical insights from recent studies provides useful context. The sample complexity analysis showing detection improves with more text is an important insight.\n\n- The discussion of open questions identifies critical challenges like evaluating detectors, improving robustness, and leveraging prompt knowledge. This will help guide future work.\n\n- Examples illustrating detection scenarios and requirements in areas like social media, academia, and customer support are insightful.\n\n**Cons**:\n\n- More comparison between the detection schemes on criteria like usability, accuracy, robustness would further highlight their tradeoffs.\n\n- While paraphrasing attacks are analyzed in depth, discussions of generative and spoofing attacks could be expanded.\n\n- The arguments around possibilities vs impossibilities could be presented in a more balanced manner, as both highlight important caveats.\n\n- More empirical evidence demonstrating the effectiveness of detection schemes on real datasets would strengthen the claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This papers provides a comprehensive survey of the nascent problem of AI-generated text detection. The survey categorizes recent work on AI-generated text detection into methods showing possibilities of detection, such as watermarking and zero-shot techniques, and methods revealing impossibilities, like paraphrasing attacks. The paper provides a comprehensive taxonomy and analysis of detection schemes, attack strategies, theoretical insights, real-world applications, and open challenges. The paper is well written and is easy to read. I enjoyed reading this paper!", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Expand the comparison between detection schemes on criteria like usability, accuracy, and robustness. A table could help summarize.\n\n- Provide more details and examples for generative and spoofing attacks. \n\n- Aim for a balanced presentation of arguments around possibilities and impossibilities. Both have merits and limitations depending on the context.\n\n- Include empirical evidence on real datasets for key detection schemes to substantiate claims of effectiveness.\n\n- Shorten repetitive details when reviewing methods to improve flow and highlight novel contributions.\n\nOverall, this is a comprehensive survey analyzing an important emerging field. Addressing the above suggestions would further improve quality and value to readers. I appreciate the authors' work and hope these comments are helpful.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong survey paper about AI-generated text detection. It gives a thorough breakdown of attacks and defenses, discusses theory and practice, and covers both "possibilities and impossibilities". The reviewers gave some suggestions for how to improve it, but uniformly found it thorough, well-structured, and well-written.
Decision: Accept (poster)