Can AI-Generated Text be Reliably Detected? Stress Testing AI Text Detectors Under Various Attacks
Authors: Vinu Sankar Sadasivan
Aounon Kumar
Sriram Balasubramanian
Wenxiao Wang
Soheil Feizi
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) can perform impressively well in various applications, such as document completion and question-answering. However, the potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concerns about their responsible use. Consequently, the reliable detection of AI-generated text has become a critical area of research. Recent works have attempted to address this challenge through various methods, including the identification of model signatures in generated text outputs and the application of watermarking techniques to detect AI-generated text.
These detectors have shown to be effective under their specific settings. In this paper, we stress-test the robustness of these AI text detectors in the presence of an attacker. We introduce recursive paraphrasing attack to stress test a wide range of detection schemes, including the ones using the watermarking as well as neural network-based detectors, zero-shot classifiers, and retrieval-based detectors. Our experiments conducted on passages, each approximately 300 tokens long, reveal the varying sensitivities of these detectors to our attacks. We also observe that these paraphrasing attacks add slight degradation to the text quality. We analyze the trade-offs between our attack strength and the resulting text quality, measured through human studies, perplexity scores, and accuracy on text benchmarks. Our findings indicate that while our recursive paraphrasing method can significantly reduce detection rates, it only slightly degrades text quality in many cases, highlighting potential vulnerabilities in current detection systems in the presence of an attacker. Additionally, we investigate the susceptibility of watermarked LLMs to spoofing attacks aimed at misclassifying human-written text as AI-generated. We demonstrate that an attacker can infer hidden AI text signatures without white-box access to the detection method, potentially leading to reputational risks for LLM developers. Finally, we provide a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. This analysis offers insights into the fundamental challenges of reliable detection as language models continue to advance.

Soheil Feizi Published in Transactions on Machine Learning Research (01/2025)
Can Al-Generated Text be Reliably Detected? Stress Testing Al Text Detectors Under Various Attacks
Vinu Sankar Sadasivan
vinu edu
Deportment of Computer Science, University of Moryland
Aounon Kumar
aokumar Citsmarr@hbs.com edu
Department of Computer Science, Harvard University
Sriram Balasubramanian
sriromb-umd. edu
Deportment of Computer Science, University of Moryland
Wenxiao Wang
www.uumd edu
Department of Computer Seience, University of Maryland
sfeizi@umd. edu
Department of Computer Science, University of Moryland
Reviewed on OpenReview: https: openreviee. net/ ptrmm? O0gg0
Abstract
Large Language Models (LLMs) can perform impressively well in various applications, such as document completion and question-answering, However, the potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concerns about their responsible use. Consequently, the reliable detection Al-generated text has become a critical area of research. Recent works have attempted address this challenge through various methods, including the identification of model signatures in generated text outputs and the application of watermarking techniques to detect Al-generated text. These detectors have shown to be effective under their specific settings. In this paper, stress-test the robustness of these AI text detectors in the presence of an attacker. We introduce recursive paraphrasing attack to stress test a wide range of detection schernes, including the ones using the watermarking as well as neural network-baser detectors, zero- shot classifiers, and retrieval-based detectors. Our experiments conducted on passages, each approximately 300 tokens long, reveal the varying sensitivities of these detectors to our attacks. We also observe that these paraphrasing attacks add slight degradation to the text quality. We analyze the trade-ofls between our atlack strength and the resulting text quality, measured through human studies, perplexity scores, and accuracy on text benchmarks. Our findings indicate that while our recursive paraphrasing method can significantly reduce detection rates, it only slightly degrades text quality in many cases, highlighting potential vulnerabilities current detection systems in the presence of an attacker. Additionally, we investigate the susceptibility of watermarked LLMs to spoofing attacks aimed misclassifying human-written text Al-generated. We demonstrate that attacker infer hidden AI text signatures without white-box access to the detection method, potentially leading to reputational risks for LLM developers. Finally, we provide theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. This analysis offers insights into the fundamental challenges of reliable detection as language models continue advance. Our code publicly available at https:/ //g1 teg.thu...... com/v www./ooowwwwnaaaaaeeeem www.saaarr//llllaallllgg. -f---ttt---------etttt---------- ---text-detector.

---

Published in Transactions on Machine Learning Research (01/2025)
LLM text detector
LLM text
LLM output
Non-LLM text
Source LLM
Input prompt
Paraphraser
Watermarked
Watermark detector
Non-watermarked Watermarked
Watermarking
output
Learn watermarking scheme
Watermarked human text (spoofing attack)
Figure 1: An illustration of vulncrabilities of existing AI-text detectors. We consider both watermarking-basee and non-watens none-wteermrkkin--beee arrow paths show the potential pipelines for adversaries to avoid detection. In red: an attacker can use a paraphraser remove the LLM signatures from an AT-generated text to avoid detection. In bluc: an adversary can query the watermarked LLM multiple times learn watermarking scheme. This information be used to spoof the watermark detector.
detectors and show that they are not reliable in practical scenarios. Colored
1 Introduction
Artificial Intelligence (AI) has made tremendous advances in recent years, from generative models in computer vision (Rombach al., 2022; Saharia al., 2022) generative models in natural language processing (NLP) (Brown et al., 2020; Zhang al., 2022; Raffel et al., 2019). Large Language Models (LLMs) can now generate texts of supreme quality with the potential in many applications. For example, the recent model of ChatGPT (OpenAI, 2022) can generate human-like texts for various tasks such as writing codes for computer programs, lyrics for songs, completing documents, and question answering; its applications are endless. The trend in NLP shows that these LLMs will even get better with time. However, this comes with a significant challenge in terms of authenticity and regulations. AT tools have the potential to be misused by users for unethical purposes such as plagiarism, generating fake news, spamming, generating fake product reviews, and manipulating web content for social engincering in ways that can have negative impacts society (Adclani et al., 2020; Weiss, 2019). Some news articles rewritten by AI have led to many fundamental errors in them (Christian, 2023). Hence, there is a need to ensure the responsible use of these generative AT tools. Tn order to aid this, a lot of recent research focuses on detecting Al-generated texts.
Recent works propose several ways, such as using trained neural network-basee detectors, zero-shot detectors, watermarking, and retrieval-based detectors for flagging AT-generated text. These detectors, especially watermarking, have shown to be effective in various settings. Neural network-based detectors approach the detection problem
binary classification task (OpenAI, 2019; Jawahar et al., 2020; Mitchell et al., 2023; Bakhtin et al., 2019; Fagni et al., 2020; Li et al., 2024). For example, OpenAl fine-tunes RoBERTa-basse detector models to distinguish between non-AI generated and GPT-2 generated texts AIpenAI, 2019). This requires such a detector to be fine-tuned with supervision on each newly released LLM for reliable detection. Zero-shot detectors address this downside of trained detectors by performing the detection task without any additional training overhead (Solaiman et al., 2019; Ippolito et al., 2019; Gehrmann al., 2019). These works evaluate the expected per-token log probability of texts and perform thresholding to detect. AT-generated texts. Mitchell et. al. (2023) observe that A AT-generated passages tend to
(Liu et al., 2019) GPT-2
2

---

Published in Transactions on Machine Learning Research (01/2025)
lie in negative curvature of log probability of texts. To leverage this observation, they propose DetectGPT, zero-shot LLM text detection method. However, zero-shot detectors require access to the original model used to generate the Al text to achieve the best performance. Additionally, neural network based and zero-shot detectors rely on a deep net for their detection, and they can be vulnerable to adversarial and poisoning altacks (Goodfellow el al., 2014; Sadasivan et al., 2023; Kumar el al., 2022; Wang et al., 2022). In comparison to these methods, watermarking significantly cases the detection of AT-generatod text by imprinting specific patterns on them that are imperceptible to humans (Atallah et al., 2001; Wilson et al., 2014; Kirchenbauer et al., 2023a; Zhao et al., 2023b; Kuditipudi et al., 2024; Zhao et al., 2023a). Soft watermarking proposed in Kirchenbauer et al. (2023a) partitions tokens into "green" and "red" lists, as they define, to help create these patterns. A watermarked LLM samples a token, with high probability, from the green list determined by a pseudo-randorn generalor seeded by its prefix loken. The watermarking detector would classify a passage with a large number of tokens from the green list as Al-generated. The soft watermarking approach of Kirchenbauer et al. (2023a) has been shown to be effective in various scttings and remains one of the popular approaches for detecting Al-generated text. For example, their watermarking scheme can achieve high true positive rate of 99.8% at false positive rate of 1% on task for classifying AI text against human- news articles. However, for watermarking to be an effective tool for preventing Al misuse, it has to be enforced on all the major LLM generators. Otherwise, an adversary could use a non- watermarking LLM for their purposes. Krishma et al. (2023) introduces an information retrieval-based detector by storing the outputs of the LLM in a database. For a candidate passage, their algorithm searches this database for semantically similar matches for detection. However, storing user-LLM conversations might cause serious privacy concerus.
Several recent news (Fowler, 2023; Hill, 2022; Das, 2023; Quach, 2023; Al-Sibai, 2023) show that some of these popular Al-text detectors fail in practical set tings. In this paper, through several experiments, we stress-test state-of-the-art Al-text detectors to evaluate their robustness in the presence of an attacker (Wolff, 2020; Aaronson, 2022; Liang et al., 2023; Pu et al., 2023; Wang et al., 2023). In $2, we have developed a recursive paraphrasing attack that use neural network-based paraphrasing to recursively paraphrase the source LLM's output lext. We perform experiments with our automated recursive paraphrasing to show the sensitivity of a range of Al text detectors to type-Il error (detecting Al text as human text). For instance, recursive paraphrasing attack tokens in length, can drop the detection rate (true positive rate at 1% false positive rate or TPR@1%FPR) from 99.3% to 9.7%. We find that our attack can add slight degradations to the Al text quality. Hence, we analyze the trade-offis between our attack and the resulting text quality, measured through human studies, perplexity scores, and accuracy of text benchmarks. Our attack differs from the relatively weaker attack from Kirchenbaner et al. (2023a) where they perform span replacement by replacing random tokens (in-place) using an LLM. Thus, our experiments show the sensitivity of the watermarking scheme against paraphrasing attacks in the presence of a stronger attacker. Zhang et al. (2024) and Lu et al. (2023) are also substitution-based attacks. Zhang et al. (2024) have a different attack objective when compared to our attack. Their attack is performed to maintain the quality of the text alone and not semantic similarity. Hence, their attack can change the original content. Lu et al. (2023) employs in-context optimization through iterative generation of word- or sentence-level substitutions using a proxy AI text detector to guide the substitutions. This positions their attack as an adversarial algorithm for evading text detection. In contrast, our approach focuses on non-adversarial itcrative or rocursive text paraphrasing attacks.
watermarked texts, even over relatively long passages of 300
After paraphrasing, the area. under the receiver operating characteristic (AUROC) curves of zero-shot detectors (Mitchell et al., 2023) drops from 96.5% to 25.2%. We also observe that the performance of neural network-based trained detectors (OpenAI, 2019) deteriorates significantly after our paraphrasing attack. For instance, the TPR& %FPR of the RoBERRTa-Large-Deeeee paraphrasing. In addition, we show that the retrieval-basee detector by Krishna et al. (2023) designed to evade paraphrase attacks is vulnerable to our recursive paraphrasing. In fact, the accuracy of their detector falls from 100% to below 60% with our recursive paraphrase attack.
from Open drops from 100% to 60% after
To quantify the drop in text quality after recursive paraphrasing, perform MTurk human evaluation studies and measure other automated metries such as perplexity and text benchmark accuracy. Our human evaluation study shows that 77% of the recursively paraphrased passages are rated high quality
3

---

Published in Transactions on Machine Learning Research (01/2025)
in terms of content preservation, and 89% of them are rated high quality in terms of grammar text quality. We also show that our recursive paraphrasing, when applied to a text benchmark such as a question-answering dataset, does not affect the performance, providing additional evidence that recursive paraphrasing does not hurt the content of the original text. Though an attacker may further improve the text quality with manual interventions, paraphrasing allacks can be sufficient. for an adversary to perform social engineering tasks such as spamming, phishing, or spreading propaganda.
Morcover, we show the possibility of spoofing attacks various AI text detectors in $3. In this setting, attacker generates non-Al text that is detected to be Al-generated, thus adversarially increasing type- error (falscly detecting human text as AT text). An adversary can potentially launch spoofing attacks to produce derogatory texts that are detected to be Al-generated to affect the reputation of the target LLM's developers. In particular, we show that an adversary can infer hidden AI text signatures without having white-box access to the detection method. For example, though the pscudo-random generator used for generating watormarked text is private, we develop an attack that adaptively queries the target LLM multiple times to learn its watermarking scheme. An adversarial humnn can then use this information to compose texts that are detected to be watermarked. Figure 1 shows an illustration of some of the vulnerabilities of the existing AI-text detectors. Gu ct al. (2024) build upon our spoofing attacks by employing watermarked data distillation to train student model to replicate the next-token distribution.
Finally, in $4, we present a theoretical result regarding the hardness of Al-text detection. Our main result in Theorem states that the AUROC of the best possible detector differentiating two distributions H (e.g., human text) and M (e.g., Al-generated text) reduces as the total variation distance TV(M, H) between them decreases. Note that this result is true for any two arbitrary distributions H and M. For example, H could be the text distribution for person group and .M could be the output text distribution of a general LLM or an LLM trained by an adversary to mimic the text of a particular set of people. Essentially, adversaries can train LLMs to mimic human text as they get more sophisticated, potentially reducing the TV distance between human and AI text, Icading to an increasingly more difficult detection problem according to our Theorem 1. Although estimating the exact TV between text distributions from a finile set of samples is a challenging problem, we provide some empirical evidence, over simulated data or via TV estimations, showing that more advancod LLMs can potentially lead to smaller TV distances. Thus, our Theorem would indicate an increasingly more difficult reliable detection problem in such cases. Our theory also indicates that if a detector becomes more robust to type-T errors, type-II errors will increase, revealing a fundamental tradeoff between type-1 and type-ll errors for the Al text detection problem. Similar tradeoffs have been explored in other domains as well. For example, Khajavi & Kuh (2016) study the relationship between detection performance and KI. divergence between input distributions the context of covariance selection. Thapliyal & Hwang (2022) show that undetectable cyberattacks can be generated by mimicking the input-output data distribution of network control systems. Although not a surprising result, Theorem is the first to link this tradeoff to the detection of Al-generated content to our knowledge.
Identifying Al-generated text is critical problem to avoid its misuse by users for unethical purposes such as plagiarism, generating fake news, and spamming, However, blindly relying on these detectors may nol be the right solution to tackle this issue since it can cause its own damages, such as falsely accusing a human of plagiarism. Our results highlight the sensitivities of a wide range of detectors to both evasion and spoofing attacks and indicate the difficulty of developing reliable detectors in the presence of an attacker. We hope to reveal the sensitivity of AT text detectors to various allacks in our stress Lests experiments.
In summary, we make the following contributions in this work.
Our work is the first to comprehensively analyze the robustness of four different classes of detectors, including watermarking- stress-test them in the presence of an allacker (in $2). In particular, the recursive paraphrasing attack that we develop is the first method that can break watermarking (Kirchenbauer et al., 2023a) and retrieval-basod (Krishna et al., 2023) detoctors. We also provide experiments to analyze the resulting text quality after our attack to find that recursive paraphrasing only leads to a slight text quality degradation in many cases.
neural network-based, zero-shot, and retrieval-based detectors, and
4

---

Published in Transactions on Machine Learning Research (01/2025)
1-1
i-2
1-3
i-4
1-5
All ppi
Content
Arg. rating
4.0 L.
4.1 1. 0.8
3.9.1.00 0.9
4.2 1. 0.9
3.7 1. 1.1
4.0 1.01100
preservation
Rasings 58c4
70.2%
77.2%
63.2%
80.0%
61.4%
70.4%
Gramuarr or text quality
Arg. rating
1.28 0.67
1.12 + 0.50
4.12 0.53
4.11 = 0.64
1.07 0.53
1.14
Ratings 58c4
87.72%
92.98%
91.23%
84.21%
89.12%
Table 1: Summary of the MTurk human evaluation study on content preservation and grammar or text quality of the recursive paraphrases with DIPPER that we use for our attacks. Ratings are on a Likert scale of to 5. See Appendix B.1 for details.
ppi
1=2
i=3
i=1
ppi
Content
Avg. rating
0.63
1.18
3.93 0.71
3.9
3.85 0.78
1.05
preservation
Rasings 58:4
91.67%
85.0%
80.0%
78.3%
80.0%
83.0%
Grammar
Arg. rating
4.62 0.55
4.28 +: 0.73
4.26
0.65
4.22 :: 0.64
1.17 0.74
4.31 0.335
text quality
Ratings 58c4
96.67%
89333396
8883399
88.3%
833333
88.0%
Table 2: Summary of the MTurk human evaluation study of the recursive paraphrascs with LLaMA-2-7B-Char
Our work is the first to show that existing detectors vulnerable against spoofing attacks where adversarial human aims to write (potentially derogatory) passage falsely detected Al-generated withoul having a white-box access to the detection methods (in $3). For instance, as proof of concept, we show that an adversary can infer the watermarking signatures by probing the watermarked LLM and analyzing the statistics of the generated tokens.
Our work is the first establish thcoretical connection between the AUROC of the bost possible detector and the TV distance between human and Al-text distributions that can be used to study the hardness of the reliable text detection problem (in $4). Our theory also reveals fundamental tradeoff between type-l and type-1l errors for the Al text detection problem.
2 Evading Al-Detectors using Paraphrasing Attacks
In this section, we first present the experimental sctup for our paraphrasing attacks in $2.1. We also provide experiments in $2.2 to study the trade-off between evasion success and text quality after the attack. $2.3 and $2.4 show the effect of the paraphrasing attacks on watermarking and non-watermarking detectors, respectively. Appendix A.l, we provide attack experiments with Alama---131 as the target model on additional detectors.
2.1 Attack Setup and Paraphrasing Methods
For evasion attacks, we consider a scenario where an adversary takes an AI response generated by a target model and then modifies the Al response in an automated and scalable fashion to evade detection. In this work, we propose the adversary modify the AI text from model using an AI paraphrascr to evade detection. Note that the adversary might be incentivized to model C. (say, watermarked) if C is powerful or might have been fine-tuned for specific applications. In these cases where L could answer a user prompt better, an adversary would still prefer to use the watermarked L to generate a response and then use a less detectable Al model P for paraphrasing to evade detection. We quantify the text quality using automated metrics such as perplexity and human studies. As shown in $2.2, our evasion attacks only lead to slight degradation in text quality while successfully evading detectors most of the lime. Note that the amount of degradation that can be tolerated is application-specifc. For example, an adversary could tolerate more quality degradation when generating a social media post than when generating a news article.
detectable
Source
LLM
output
Paraphraser
recurse I-1 times
ppi
Figure 2: Recursive paraphrasing
5

---

Published in Transactions on Machine Learning Research (01/2025)
We use the "document" features of the XSum dataset (Narayan et al., 2018) con- taining 1000 long news articles (~~300 tokens in length) for our experiments. In Appendix A, we perform experiments with additional datasets - a medical text dataset Pub.MedQA (Jin al., 2019) and dataset with articles from 10 different domains Kafkai (Kafkai, 2020). As target LLMs, we use O.T-L.3B and OPT-13B (Zhang et al., 2022) language models with 1.3B and 13B parameters, respectively. In Appendix A, we also evaluale our attacks with GPT-2 Medium (Radford el al., 2019) as the target model. We use three different neural betwork-based paraphrasers - DIPPER with 11B parame- ters (Krishna et al., 2023),
Paraphraser
Evaluation
AT text
pp1
Pp2
Pp3
ppp4
pp5
DIPPER
Mesu perplesity
5.2
7.7
7.8
8.5
7.7
8.7
QA performance
97%
97%%
96%
969
96%
95.3%
LLMA--2-7B- Clat
Moan perploiit
6.2
9.3
10.3
10.5
QA Performance
97%
97%
97%
97%
97%
97%
Table 3: Automated evaluation of the text quality of recursive paraphrases using perplexity mcasures with respect to OPT-13B and questtin-ansswringg benchmark accuracy.
with 7B paramcters (Touvron et al., 2023), and T5-basod paraphraser (Damodaran, 2021) with 222M parameters. Suppose passage S (81, 82, ..., 8a) where 8, is the 2th sentence. DIPPER and LLaMA-2-TH 2-B------B----- 1B-Chat paraphrase S to be S'= fstrong (S) in one-shot while the light-weight T5-based paraphraser would output 8' = ( fweak(52)) .... fweak (sn)) where they can only paraphrase sentence-byy be-ee-----eeence.. DIPPER and 2-7M-A---BB-haa also have the ability to input a context prompt text C to generate higher-quality paraphrasing S' = Sstrong (S, C.). We can also vary two different hyperparameters of DIPPER to generate a diverse number of paraphrases for a single input passage.
We use DIPPER and 2 2-t- for recursive paraphrasing attacks since they provide high-quality paraphrasing when compared to the 222M parameter T5 model. Let an LIM L generate AT text output 5 L(C for an input prompt C. DIPPER or LLaMA-2-7B-Chat can be used to generate paraphrase pp1(S) = [atrang(S, C). This paraphrasing can be performed in recursion (see Figure 2). That is, pp2(S)= = fstrong (p C) and so on.
While DIPPER is explicitly trained to be a paraphraser, LLaMA-2-7B-Chat is an instruction-tuned model for chat purposes. We design a system prompt (see Appendix B.2) with the LLaMA-2-7B-CHA model to use it as a paraphraser. In $2.3 and $2.4, we show that recursive paraphrasing is effective in evading the strong watermark and retrieval-based detectors when compared to a single round of paraphrasing. Using human and other automated evaluation techniques in $2.2, we show that our recursive paraphrasing method only degrades the text quality slightly most of the time.
2.2 Quality of the Paraphrases
In order to reliably study the quality of the recursive paraphrases we use in our experiments using DIPPER and LLaMA-2-7B-Chat we perform human evaluations using Mllurk and other automated techniques. The Al-text used in this study is generated using a watermarked OPT-13B model. Tables 1 and 2 provide a summary of the study. We investigate the content preservation and text quality or grammar of the recursive paraphrases with respect to the Al-generated texts (see Tables 7-10 in Appendix B.1 for more details). In terms of content prescrvation with DIPPER, 70% of the paraphrases were rated high quality and 23% somewhat equivalent. In terms of text quality or grammar, 89% of the paraphrases were rated high quality. On Likert scale of 1 to 5, the DIPPER paraphrases that we use received an average rating of 4.14 + 0.58 for text quality or grammar and 4.0 + 0.9 for content preservation. Similarly, 83% of the recursive paraphrases we obtain with LLaMA-2-7B-Cha were rated high quality. See Appendix B.1 for more details on the human study.
For automated text quality evaluations, we use perplexity measures and a question-answering (QA) benchmark in Table 3. We measure the perplexity scores using OPT-13B. As shown in the table, the perplexity scores degrade from 5.5 to 8.7 and 10.5, respectively, for DIPPER and MaMA--2-B--Cha after 5 rounds of paraphrasing. We also use a QA benchmark SQuAD-v2 (Rajpurkar et al., 2016) to evaluate the effect of recursive paraphrasing. For this, we use two hundred data points from SQuAD-v2, which have context text length of at least 300 tokens. The length of context text passages we use in the study is 328 + 28 tokens. Each data point consists of a context passage, a question, and an answer. We evaluate QA model on the SquAD-v2 benchmark observe that it achieves 97% accuracy in the QA task. For the QA model, we use the M-MMA-2-13B- 13B-Chat model with carefully written system prompt (see Appendix B.2). To evaluate the quality
6

---

Published in Transactions on Machine Learning Research (01/2025)
1.0
1.0
0.8
0.8
0.6
0.6
TPR
TPR
Watermarking AUROC:O 999, 1PASINIPL- 998, pph:51
0.4
Watemarking AUROCIO. 999, TRRISSSEPRRO.OO po0s
pol
AUROC TPR PPPPP 807, 7.77.7
R61
AUROC:n 906, TPPU 1AAG1TAARR 11/PR-O 146,, ppin.1
AUROC:0.D5S TPR@1S/PR
0.2
pa3
AUROC.O.SS 853,, TPR@: SLFI:O
ppk.8.5
0.2
pp3
AUROC:0.923, O
5pt9.0
P34
AUROCIO 892, TPR@1NEPR: 227, p.77.7
Rp4
AUROC: 808. TIPi TARGI1 1NFFFIL.. 0.266 ppl:10.3
AUROC.0.76. (PR@1NFPRRDD.
pok.8.
AUROCIO.P9 1PR@11/PP
ppl:10.5
ppl
AURDC.0.612, 1PNg18FF9.0.00
ppha..
Best of
AUROC/0.738, TFR:G115FFRO ppl:11.5 0.00
0.0
Random
AUROC:0.300 TPR@1% PR:0.000
0.0
.....
Random
AUROC. 0.5000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
FPR
(a) Recursive paraphrasing with DIPPER
(b) Recursive paraphraasing with LLaMA-2-7B-Chai
Figure ROC plots for soft watermarking with recursive paraphrasing attacks. AUROC, TPR@1%EPR, and perplexity scores measured using OPT-13B are given in the legend. The target LLM OPT-13B is used generate watermarked output that are 300 tokens in length.
of paraphrases, we paraphrase the context passages recursively and use these to evaluate the QA accuracy with the QA model. the QA model can answer the question correctly based on the paraphrased context, then the information is preserved in the paraphrase. As we observe, the QA performance with recursive paraphrasing is similar to that with the clean context passage. These results substantiate that Al text detectors can be effectively attacked using recursive paraphrasing with only slight degradation in text quality.
We note that the amount of acceptable quality degradation can be application-speeifi. For example, an adversary might be okay with a higher quality drop when writing a social media post than when writing a fake news article. Our human studies rate our recursive paraphrases to have a score of either or 4 over 70% of the time. Though this might be acceptable for some adversaries, others might not tolerate a score of for their applications. Since a score denotes minor degradation, we presume that the adversarics could manually edit. them for their attacks. Nevertheless, our paraphrases get perfect score 35% of the time, indicating that it is still practical for adversaries to use it to perform their attacks successfully. However, we believe this tradeoff in text quality degradation and detection evasion would diminish as paraphrasers improve in the future.
2.3 Paraphrasing Attacks on Watermarked Al Text
In this section, evaluate our recursive paraphrasing attacks the soft watermarking scheme proposed in Kirchenbauer et al. (2023a). Soft watermarking encourages LLMs to output token s(t) at time-step that belongs to "green list". The green list for s(0) is created using private pseudo-random generator that is seeded with the prior token g(t-1), A watermarked output from the LLM is designed to have tokens that are majorly selected from the green list. Hence, a watermark detector with the pseudo-random generator checks the number of green tokens a candidate passage detect whether it is watermarked or not. Here, we target watermarked OPT-13B with 13B parameters Figure 3 and watermarked OPT-1.3B in Figure for our experiments. In Appendix A.2, we also evaluate our atlacks on GPT-2 Medium (Radford et al., 2019) and other datasets PubMedQA (Jin et al., 2019) and Kafkai (Kafkai, 2020).
Datasct. We perform our experiments on 2000 text passages that are around 300 tokens in length (1000 passages per human and AI text classes). We pick 1000 long news articles from the XSum "document" fealure. For each article, the first ~300 tokens are input to the target OPT-1.3B to generate 1000 watermarked AI text passages that are cach <300 tokens in length. The second 300 tokens from the 1000 news articles the dataset are treated as bascline human text. We note that our considered dataset has more and longer passages compared to the experiments in Kirchenbauer et al. (2023a).
7

---

Published in Transactions on Machine Learning Research (01/2025)
1.0
1.0
0.8
0.8
0.6
0.6
TPR
TPR
Watermanding tokens- AUROC:0.996, TRRg15FFR.D.0
0.4
Watermarting AUROC:0.998,
TMRg15FPh:0093 ppt:7.6
0.4
Best of ppl
tokann- 50 AUROC:00436, TPAg1SSFPR.C.OOM
991
AUROC:0.755
TFRD11/FPR 786, ppt:0.3
Watermarking tolerss=10 AUROC:0.990 TPR@15EPRO.500
AUROC.0.917,
Best pot
tokans-100 AUROC:0.518, 1PRG196FPE00007
0.2
PD4
P01
AUROC:0.004 804, TFR@155FRO.014
AUROC-0.840
TPR@ 1PSPR.00000222
ppt
ppla.7
0.2
Watermarking tokens-200 AUROC:0 996, IPA@INFPRD 992
AURDC:0.611, TPRg19FFR-00044
AUR0C-6.738, TPh0115FPRC0000
Watermarking
tokerss300 AUROC.0. 998, TPA@ISEPPED.D0
Best op!
AUROC:0.000 0oc.
TPR@I9/PRO..C 040,
ppt:9.1
Best of ppl
AUROC:0.000 TPR PR:0.040 tokens=200
0.0
Random
AUROC.0.500,
1FR:115FPP.0.000
0.0
1PM015FFM000000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
FPR
(a) Watermarked text with mean token length 300
(b) Watermarked text with varying token lengths
Figure ROC plots for soft watermarking with recursive paraphrasing attacks. AUROC, TPR@1 %FPR, and perplexity scores measured using OPT-13B are given in the legend. The target LLM OPT-1.3.. (a) Even for 300 tokens long watermarked passages, recursive paraphrasing is effective. As paraphrasing rounds proceed, detection rates degrade siguificantly with slight. trade-off in text quality. (b) Attacking watermarked passages become easier as their length reduces.
1.0
True Positive Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
10-1
10-2
10-1
10-1
10 False Positive Rate
10-1
100
10-3
10-2
10-1
10
wadee 07. 80,, (.1)
putt Ootfeee J.z (0U99 5806, E
pert.ibeton 38,4 10.59. 3.04,
Wieihood, 3 Eireshold 10 11A 0,03 0.00
Isg, mak threchoie (C. 6.6)
roberta base ope detectorr (0 3E1, 8.105, 0.071
intirnsdee 10
pertuihertes 38,2 325 0.813.
mik, peteehesttal oli 10, 864, 0.00 0.01
enthopyy frresnale 23306, 6.433, ann roberta-lane www.soppeaaeeeeee comalddetactor (3 0.923, 0.792)
Figure 5: ROC curves for various trained and zero-shot detectors. Left: Without attack. Middle: After paraphrasing atlack using T5-based paraphraser. The performance of zero-shol delectors drops significantly. Right: Here, we assume we can query the detector ten times for the paraphrasing attack. We generate ten paraphrasings for cach passage and query multiple times to evade detection. Notice how all detectors have low T 1%FPR. In the plot legend perturbation refers the zero-shot methods in Mitchell et al. (2023); threshold refers to the zero-shot methods in Solaiman et al. (2019); Gehrmann et al. (2019); Ippolito et al. (2019); roberta refers to OpenAl's trained detectors (OpenAl, 2019). The TPR01%FPR scores of different detectors before the attack, after the attack, and after the attack with multiple queries, respectively, are provided in the plot legend.
Detection results after paraphrasing attack. Weaker paraphrasing attacks discussed in Kirchenbauer al. (2023a) are not effective in removing watermarks. They perform "span replacement" by replacing random tokens (in-place) using language model. However, after single round of paraphrasing (pp1) with watermarked OPT-13B as the target LLM, TPR 1%FP of watermark detector degrades from 99.8% to 80.7% and 54.6%, respectively, with DIPPER and I.LMAA2-7B-Chhm paraphrasors. Though watermarking is a worthwhile endeavor to prevent AI plagiarism, with our stress lest, we show that an adversary can find their way to evade detection via paraphrasing. As shown in Figures 3-4, the recursive paraphrase attack further degrades the dctection rate of the detector to below 20% after 5 rounds of paraphrasing (pp5). Note that all the settings pp2 or 2 rounds of paraphrasing sufficient to degrade TPR@1 %FPR to below 50%. As shown in Figure 3, DIPPER, shows clearer and more consistent trend in improving attack performance recursions
8

---

Published in Transactions on Machine Learning Research (01/2025)
of paraphrasing in comparison to LLaMA-2. This is because DIPPER is trained explicitly to be a paraphraser with hyperparameters that can control the quality of paraphrasing. Therefore, we mainly employ DIPPER for our recursive paraphrase attacks. Best of ppi in the figure refers to the method where, for each passage, we select the paraphrasc out of all the ppi's that has the worst detector score. For Best of ppi with OPT-133, the detection rale reduces drastically from 99.8% to 4.0% with trade-off of 1.5 in the perplexity score (Figure 4a). Best of ppi, unlike the ppi attacks, assume black box query access to the detector. Figure 4b shows that the watermarking detector becomes weaker as the length of the watermarked text reduces. Note that for watermarked texts that are 50 or 100 tokens long, the detection performance after the recursive paraphrasing attack is similar to that of a random detector. As the plot indicates, watermarking could be more reliable for preventing Al plagiarism in tasks that require longer texts. However, this does not guarantee that watermarking will be a soolproof defense in such settings. This requires more investigation, and we leave this for future work. We provide examples of paraphrased text that we use for our attacks in Appendix B.3.
2.4 Paraphrasing Attacks on Non-Watermarked Al Text
Neural network-based trained detectors such as SarreTa--arge-eeeee trained or fine-tuned for binary classification with datasots containing human and AT-generated texts. Zero-shot classifiers leverage specific statistical properties of the source LLM outputs for their detection. Retrieval-based methods search for candidate passage in a database that stores the LLM outputs. Here, perform experiments on these non-watermarking detectors to show they are vulnerable to our paraphrasing attack.
from OpenAT (OpenAT, 2019) are
Trained and Zero-shot detectors. We use a pre-trained GPT-2 Medium model (Radford et al., 2019) with 355M parameters as the target LLM to evaluate our attack on 1000 long passages from the XSum dataset (Narayan et al., 2018). We use the T5-basod paraphrasing model (Damodaran, 2021) with 222M parameters to rephrase the 1000 output texts generated using the target GPT-2 Medium model.
Figure 5 shows the effectiveness of the paraphrasing attack over these detectors. The AUROC scores of DetectGPT (Mitchell et al., 2023) drop
7.6
100
8.3
Detection performance (%)
8.7
8.7
from 96.5% (before the attack) to 59.8% (after the attack). Note that AUROC of 50% corresponds to a random detector. The rest of the zero-shot detectors (Solaiman et al., 2019; Gehrmann al., 2019; Ippolito et al., 2019) also perform poorly after our attack. Though the performance of the trained neural network-based detectors (OpenAT, 2019) is better than that of zero-shot detectors, they are also not reliable. For example, TPR@1%FPR of Ope- nAI's RoBERTa-Large Detector Letteeectorr drops from 100% to around 92% after our attack.
80
8.9
9.8
9.3
40
Retrieval-basee detector
Random detector
No
pp1
pp2
pp3
pp4
pp5
Best
attack
ppi
In another setting, we assume the attacker may have multiple access to the detector. That is, the attacker can query the detector with an input AT text passage, and the detector would reveal the detection score to the attacker. For this scenario, we generate ten different paraphrases for an input passage and query the detector for the detection scores. For each AI text passage, we then select the paraphrasc with the worst detection score for evaluating the ROC curves. As shown in Figure 5, with multiple queries to the detector, an adversary can paraphrase more efficiently to bring down TPR@17FPR of the RoBERTa-Large-Dettet from 100% to 80%. In Appendix A.3, we show more experiments with more datasets and target LLMs.
Figure 6: Recursive paraphrasing breaks the retrieval- based detector (Krishna et al., 2023) with only slight degradation in text quality. ppi refers to i recursion(s) of paraphrasing. Numbers next to markers denote the perplexity scores of the paraphraser output..
As seen in the results, the detection of the entropy threshold detector improves with paraphrasing. LLMs are trained on human-written texts, and for this reason, they might have low entropy scores on human-written samples we use in our experiments due to memorization. Therefore, the entropy detector might have poor detection scores before the paraphrasing attack. However, after paraphrasing with a different AT model, the
9

---

Published in Transactions on Machine Learning Research (01/2025)
entropy scores for these human-writter samples might increase, improving the detection scores. Despite this, the entropy threshold detector has poor detection rates before and after the attack.
Though the performance of performance of trained detectors degrades after each round of paraphrasing, they seem to be more robust to paraphrase attacks than the other detectors we study. We hypothesize that this might be due to these detectors being trained on human-written samples we use for our study. For example, the MAGE dataset (Li et al., 2024) includes passages from the XSum dataset we use. Gameiro et al. (2024) argues that while trained detectors can generalize better to unseen LLMs, they may overfit to this training distribution of human text. They also show that some of these detectors fail to generalize to out-of-distribution numan-writton text. This is an aspect that we do not consider in our work, but would still make these detectors unreliable for real-world applications.
Retrieval-based detectors. Detector in Krishna et al. (2023) is designed to be robust against paraphrase attacks. However, we show that they can suffer from the recursive paraphrase attacks that we develop using DIPPER. We use 2000 passages (1000 generated by OPT-1.3B and 1000 human passages) from the XSum dataset. AI outputs are stored in the AI database by the detector. As shown in Figure 6, this detector detects almost all of the Al outputs even after a round of paraphrasing, However, the detection accuracy drops below ~ 60% after five rounds of recursive paraphrasing. As marked in the plot, the perplexity score of the paraphrased text only degrades by 1.7 at a detection accuracy of ~ 60%. Moreover, retrieval- detectors are concerning since they might lead to serious privacy issues from storing users' LLM conversations. In Appendix A.4, we show more experiments with more datasets and target LLMs.
3 Spoofing Attacks on Generative Al-text Models
An AI language detector without low typo-I error can cause harm as it might wrongly accuse a human of plagiarizing using an LLM. Moreover, attacker (adversarial human) can generate non-AT text to be detected as Al-generated. This is called the spoofing ottack. An adversary can potentially launch spoofing attacks to produce derogatory texts to damage the reputation of the target LLM's developers. In this section, as a proof-of-concept, we show that current lext detectors can be spoofed to detect texts composed by adversarial humans as AT-generated. More details on the spoofing experiments are presented in Appendix D.
Soft watermarking. As discussed in $2, soft watermarked LLMs (Kirchenbauer et al., 2023a) generate tokens from the "green list" that are determined by a pseudo-random generator seeded by the prefix token. Though the pseudo-random generator is private, an attacker can estimate the green lists by observing multiple token pairs in the watermarked texts from the target LLM. An adversarial human can then leverage the estimated green lists to compose texts by themselves that are detected to be watermarked. In our experiments, we estimate the green lists for 181 most commonly used words in the English vocabulary. We query the target watermarked OPT-1.3B model one million times to observe the token pair distributions within this smaller vocabulary subset we select. Note that this attack on a watermarked model only needs to be performed once to learn the watermarking pattern or the proxy green list to spoof it thereafter. Based on the frequency of tokens that follow a prefix token in the observed generative outputs, each of the 181 common words. We build tool that helps adversarial humans create watermarked sentences by providing them with the proxy green list that we learn with only access to a watermarked text corpora obtained from the target watermarked LLM. We observe that the soft watermarking scheme can be spoofed to degrade its detection AUROC from 99.8% to 1.3% (see Figure!! 7).
estimate green lists for
True Positive Rate
1.0
Vistermmn king
Spoofing
rendom
0.8
0.6
0.4
0.2
0.0
0.0
0.7
0.4
0.6
0.8
1.0
False Positive Rate
Retrieval-based detectors. Krishna et al. (2023) use database to store LLM outputs to detect AI-text by retrieval. We find in our experiments (see Figure 13) that an adversary can spoof this detector 100% of the time, even if the detector maintains a private database. Suppose an ad- versary, say teacher, has access to human written document S, say student's essay. The adversary
Figure ROC curve of a soft watermarking based detector (Kirchenbauer et al., 2023a) after spoofing attack.
10

---

Published in Transactions on Machine Learning Research (01/2025)
prompt the target LLM to paraphrase S to get S'. This results in the LLM, by design, storing its output 5 in its private database for detection purposes. Now, the detector would classify the original human text S as Al-generated since a semantically similar copy S' is present in its database. In this manner, a teacher can purposcfully allege an innocent student to have plagiarised using the retrieval-based detector. Note that manipulating retrieval-based delectors is easier using this approach compared to watermarking techniques. This observation implics a practical tradeoff between type-I and type-II errors. When a detector is strengthened against type-ll errors, it tends result in a deterioration of its performance in terms of type-I errors.
Zero-shot and neural network-based detectors. In this sctting, a malicious adversary could write a short text in a collaborative work, which may lead to the entire text being classified as Al-generated. To simulate this, we prepend a human-writtee text marked as Al-generated by the detector to all the other human-generated text. for spoofing. In other words, from 200 long passages in the XSum dataset, we pick the human text with the worst dctection score for each detoctor considered in $2.4. We then prepend this text all the other human texis, ensuring that the length of the prepended text does not exceed the length of the original text. Our experiments show that the AUROC of all these detectors drops after spoofing (see plots in Appendix D). After this naive spoofing attack, the TPR01%FPR of most of these detectors drop significantly.
4 Hardness of Reliable AI Text Detection
In this section, we formally upper bound the AUROC of arbitrary detector in terms of the TV between the distributions for M (e.g., AI text) and H (e.g., human text) over the set of all possible text sequences Ω. We note that this result holds for any two arbitrary distributions H and M. For example, H could be the text distribution for a person or group, while M could be the output text distribution of a general LLM or an LLM trained by an adversary to mimic the text of a particular set of people.
We use TV(M, H) to denote the TV between these two distributions and model detector as a function D:Q-4 R that maps every sequence in Q to a real number. Sequences are classified into Al-generated or human-generated by applying a threshold 7 on this number. By adjusting the parameter y, we can tune the sensitivity of the detector to Al and human-generate
texts to obtain ROC curve.
Theorem 1. The area under the ROC of any detector D bounded as
1 + TV(M, TV(M, H)2
AUROC(D) S 2
2
The proof is deferred to Appendix C.1. Figure 8 shows how the above bound grows as a function of the TV distance. This theorem states that as the TV distance between Al and human text distributions reduces, the AUROC of the best possible detector decreases. Based on our theory, an adversary can use advanced LLMs to mimic human text to reduce the TV distance between human and Al text distributions to evade text detection systems.
0.9
0.8
Best Possible
aUROC
0.6
0.5
Random Classller
0.3
For a detector to have good performance (say, AUROC > 0.9), the distributions of human and Al-generated texts must be very different from each other (TV > 0.5 based on the figure). As M gets more similar to H. (say, TV < 0.2), the performance of even the best-possible detector becomes unroliable (AUROC < 0.7). For some applications, say Al-text plagiarism, reliable detection should have a low false positive rate (say, < 0.01) and a high true positive rate (say, > 0.9). Based on our theory, this cannot be achieved even when the overlap between the distributions is relatively low, say 11% (or TV 0.9 0.01 0.89, based on equation 1 in Appendix C.1).
0.2
0.1
0.0
0.8 0.9 10
Total Variation
Figure 8: Comparing the performance,
terms of AUROC, of the best possible de- tector to that of the baseline performance corresponding to a random classifier.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Sequence length
0.40
Total Variation Estimate
GP1-2-5
GPT-2-4
Total variation
0.35
0.20
G7T-2-M
GPT-2-XL
0.30
0.25
0.15
0.20
0.10
0.15
dimension
0.10
0.05
24
0.05
333
0.00
Vocabulary size
4
5
Sequence Length
Figure 9: Increasing model size reduces the exact TV between the true synthetic data distribution and the learned distribution. Error bars report standard deviations after 5 independent trials.
Figure 10: Estimated TV distances of GPT- 2 output datasets from the WebText dataset using meta-token sequences of varying lengths. TV decreases with model size for each length.
Note that, for watermarked model, the above bound can be close to one as the TV between the watermarked distribution and duman--eneratee distribution can be high. Corollary 1 in Appendix C.2 discusses how paraphrasing attacks can be effective in evading watermarks using Theorem 1. In Appendix C.3, we also present a tightness analysis of the bound in Theorem 1, where we show that for any distribution H there exists M and a detector D for which the bound holds with equality. We also discuss general trade-ofTs between true positive and false positive rates of detection in Corollaries and 3 in Appendix C.2. Theorem 2 in Appendix C.4 extends Thcorem to bound the AUROC of the best possible detector by a function of the TV distance between LLM outputs generated using pseudorandomness and human text distributions.
In studying the hardness of the detection problem, we consider the following assumption that for given human-text distribution H. more advanced LLMs mimicking H. can lead to smaller TV. Thus, using Theorem 1, the detection problem becomes increasingly more difficult. This is the core argument of our hardness result. on AT text detection. Although the underlying assumption seems to be intuitive given the capabilities of LLMs such as GPT-4 (OpenAI, 2023), a precise analysis of this assumption is quite difficult because estimating the true TV of the text distributions from a finite set of samples extremely challenging. Nevertheless, we provide some empirical evidence supporting this assumption using two sets of experiments. In all the experiments, consistently observe that the TV distance estimates between human and AI text distributions reduce as language models gel more advanced, indicaling the increasing difficulty associated with AI texi detection.
(i) Using synthetic text data. We perform experiments on a toy synthetic text dataset where the exact TV distance can be calculated. We use the Markov assumption to generate the synthetic text data with sequence length using a randomly generated loken transition matrix for varying vocabulary sizes. We use single-layer LSTMs of different hidden unit sizes to train on a dataset of size 20,000 sampled from this synthetic data distribution using a default AdamW optimizer (Loshchilov & Hutter, 2017). We compute the learned token transition matrix for the LSTM output distribution using the softmax logit values of the trained model. Using transition matrices of both distributions, we compute the exact TV. Figure 9 shows that the exact TV distances between the learned and true synthetic distributions reduce as the LSTM model size increases.
(ii) Using projection. For discrete distributions, the TV distance can be computed as 1/2 of the sum of the point-wise differences between their probability density functions (PDFs). While this is mathematically simple since texts can be considered as token sequences with bounded length, it is not practical to compute true TV distances directly through estimating PDFs due to the size of the sample space, which is approximately the size of the token set to the power of sequence length. To tackle this issuc, we split the original token set into five roughly equal partitions and assign a mela-token to each partition. Given a sequence of Lokens from the original set, we construct a new sequence by replacing each token with the corresponding meta-token. We estimate the PDFs of the sequences of meta-tokens created using texts from the WebText and GPT-2 output datasets. Since the set of meta-tokens is significantly smaller than the original token set, estimating PDFs becomes much more tractable. We then use these PDFs to estimate the total variation distances of the output.
12

---

Published in Transactions on Machine Learning Research (01/2025)
distributions of different GPT-2 models 2GPll--Small, GPT-2-Mediume GPT-2-Large, and GPT-2-XL) from the WebText dataset. Figure 10 plots these TV estimates for different sequence lengths, averaged over 30 runs of the experiment. We observe that the TV distance consistently decreases with increasing model size for all sequence lengths.
These experiments provide empirical evidence that more advanced LLMss can lead smaller TV distances. Thus, based on Theorem 1, reliable Al text detection would become increasingly difficult.
Acknowledgments and Disclosure of Funding
This project was supported in part by NSF CAREER AWARD 1942230, ONR. YIP award N00014-22-1-2271, NTST 60NANB20D134, Meta award 23010098, HR00111955 11900066 (GARD), Army Grant No. W911NF2120076, a capital one grant, and the NSF award CCF2212458 and an Amazon Rescarch Award. Sadasivan is also supported by the Kulkarni Research Fellowship. The authors would like to thank Keivan Rezaei and Mehrdad Saberi for their insights this work. The authors also acknowledge the use of OpenAl's ChatGPT to improve clarity and readability.
References
Scott Aaronson. My safoty lecture for ut offective altruism. November 2022. URL https::/ssotttaaooss.. blog/?p=6823.
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. Generating sentiment- sentiment-preserringg fake online reviews using neural language models and their human-an bochino-bbased detection. In Advanced Information Networking and Applications: Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA-2020), pp- 1341 1354. Springer, 2020.
Noor Al-Sibai. Ai plagiarism detection software keeps falsely accusing students of cheating. Futurism, 2023. URL
cww/como/a////a ai pplaaiarisss lalaagiarrmmmsssttrre---atae fagttrrrr--l
Mikhail J Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of-concepl implementation. In Information Hiding: 4th International Workshop, TH 2001 Pittshurgh, PA, USA, April 25 27, 2001 Proceedings 4, pp. 185 200. Springer, 2001.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Mare' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906 100i0.9900.0351,,,, 2019.
Lenore Blum, Manuel Blum, and Mike Shub. Comparison of two pseudo-random mumber generators. Advances Cryptology: Proceedings of CRYPTO '82, pp. 61-78. Plenum, 1982.
Manuel Blum and Silvio Micali. How to generate eryptographically strong sequences of pseudorandom bits. SIAM Journal on Computing, 13(4):850 864, 1984. doi: 10.1137/021305) URL https://doi.org/10
1137/0213053.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020.
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection, 2023.
Jon Christian. Cnet secretly used ai on articles that didn't disclose that fact, staff say. January 2023. URL https://futurism. wttprr/sw..ooceeeetaaaarrrteeeee antae
Prithiviraj Damodaran. Parrot: Paraphrase generation for nlu, 2021.
13

---

srrrudeeteeceeatttnn--fotaaeeer Published in Transactions on Machine Learning Research (01/2025)
Mehul ing ai. p cestector-turnitiie teeesiinn-kee--eee t html.
Reuben
Das. Turn-it-in: Firstpost, 2023.
Ai
fails students for not
URL http::/wwwwwwrsppssttooo https/ooott.oooooo//////
117144666..
T Fagni, F Falchi, Gambini, A Martella, and M Tesconi. Tweepfake: About delecting deeplake tweets. arxiv, arXiv preprint arXiv:2008.00 00036, 2020.
Geoffrey A. Fowler. We tested new chatgpt-detector for teachers. it flagged innocent stu- dent. The Washington Post, 2023. URL https://www washingtonpost. hetere---haai---ttttiro------ee
http:://wwwwwaahnggggoo.oo. com//eecmoologg//20///////000
Henrique Da Silva Gameiro, Andrei Kucharavy, and Ljiljana Dolamic. Lim detectors still fall short of real world: Case of IIm-generated short news-like posts, 2024. URL https///aaxx..oor//////22... 03291.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arxiv:1966, 04043, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint orXXf:1412.600 2014.
Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models, 2024. URL https://arxiss wwwaaiii...//abbb. wttg/aa///12.../00 04469.
Kashmir Hill. Accusod of cheating by an algorithm, and a profcssor she had never met. The New York Times, 2022. URL https://sait www..nytimes...m wwtm/20/2///////////m///// /ttpa////////eeecoooogo///// html.
Daphne Tppolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. ar preprint arXiv:1911.0000 2019.
Ganesh Jawahar, Muhammad Abdul-Mageod, and Laks VS Lakshmanan. Automatic detection of machine generaled lext: A critical survey. arXiv preprinl ar arX:::2011.01314 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint. Conference on Natural Language Processing (FMNIP--JCNT.P)) pp. 2567-2577 Hong Kong, China, November 2019. Association for Computational Linguisties. doi: URL https://aclantholoog) www.:/loooyoorg/1-11/000 1229
Kafkai. "kafkai: Ai writer & ai content generator". 2020. URL https://kafkai.comm
Navid Tafaghodi Khajavi and Anthony Kuh. The quality of the covariance selection through detection problem and AUC bounds. CoRR, abs/ abs/1605.0577/ 2016. URL
05776.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint or orXiu:2301. 10226 2023a.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models, 2023b.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Tyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-freo watermarks for language models, 2024. URL https: http://aav....///////0/0/0/// wrgg/bs//30...oo/b./
15593. 155933
14

---

Published in Transactions on Machine Learning Research (01/2025)
Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Certifying model accuracy under distribution shifts. arXiv preprinl or PrXx::2001.12400 2022.
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Mage: Machino-generater text detection in the wild, 2024. URL https://arxiv..rg 13242.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt. delectors are biased against. non-native english writers, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprinl Xiv: (00::100..0 1692, 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.050 2017.
Ning Lu, Shengcai Liu, Rui Hc, Qi Wang, Yew-Soon Ong, and Ke Tang. Large language models can be guided lo evade ai-generated lext detection. arXiv preprint urXie:2305. 10847, 2023.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint 20X::2301.11305, 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv,
2018
OpenAL. Gpt-2: 1.5b release. November 2019. URL
OpenAL. Chatgpt: Optimizing language models for dialogue. November 2022. URL http:https:/oooea..ccm blog/chatgpt/.
OP Gpt-4 technical report. March 2023. URL https://ccnn.opanne wwt.:/pwww..ppppp.../eee /ptp/ 4.pff
J. Pu, Sarwar, Abdullah, A. Rehman, Y. Kim, P. Bhattacharya, M. Javed, and B. Viswanath. Deepfake text detection: Limitations and opportunities. In 2023 IEFF Symposium on Security and Privacy (SP), pp. 1613 1630, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: D0.1109/SP463 20233. 10179387. URL attps://dat ieeecompttersocieeeeeoo.oo wttp/....../////...... 0091//9/////...... 2023. 10179387.
Katyanna Quach. Professor freezes student grades after chatgpt claimed ai wrote their papers. The Register, 2023. URL https: //wp:. http://eeeeggeeee..o///////////0
cchatpttgradess////
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and 1lya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2019. URL attps://arxx
argg/bss/9110 10683.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. MwXiv:1606.05250, 2016.
Robin Rombach, Andreas Blattmamn, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Vinu Sankar Sadasivan, Mahdi Soltanolkotabi, and Soheil Feizi Cuda: Convolution-based unlearnable datasets. arXio preprint ar Xrxi:23303.0423 0420 2023.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11455 2022.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908. 10ee:9000090,,, 2019.
Omanshu Thapliyal and Inscok Hwang. Data-driven cyberattack synthesis against network control systems, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXic preprint arXiv:2307.0995 092888 2023.
Wenxiao Wang, Alexander
J Levine, and Soheil Feizi. Improved certified defenses against data poisoning with (deterministic) finite aggregation. In International Conference on Machine Learning, pp. 22769 22783. PMLR, 2022.
Yuxia Wang. Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, and Preslav Nakov. M4: Multi-generator, multi- multi-domain, and multi-lingual black-box machine-generated text detection, 2023.
Max Weiss. Deepfake bot submisssion to federal public comment websites cannot be distinguished from human submissions. Technology Science, 2019121801, 2019.
Alex Wilson, Phil Blunsom, and Andrew Ker. Linguistic steganography on twitter: hierarchical language modeling with manual interaction. In Media Watermarking, Security, and Forensics 2014, volume 9028, pp. 9-25. SPIE, 2014.
Max Wolff. Attacking neural text detectors. CoRR, abs/ 21022 2020. URL 1006811768.
/rg/abs/
Hanlin Zhang, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models, 2024. URL
https://arxiv http://xxxvvvv..oaaaa//// hts/abb//230/ 22111.03378.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL ptttps://arxti org/ httg/abb/2205.0000 01068.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-genorated text, 2023a. URL https: http//rrxvv..rgaaeeee/a org/abs/2300 17439.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible water- marking. arXiv preprint 2tw:2302.03122 2023b.
A Experiments with More Datasets and Models
In this section, consider multiple datasets (XSum (Narayan et al., 2018), PubMedQA (Jin et al., 2019), and Kafkai (Pu et al., 2023)) and target LLMs (OPT-1.3B (Zhang et al., 2022) and GPT-2-Medium (Radford et al., 2019)) for analyzing our attacks.
Datasets. As discussed in the $ 2.1, we use 2000 text passages (1000 passages each for human and AI- generated text classos) of 300 tokens length from the XSum dataset for analyzing our attacks. For the rest of the datasets, we use 1000 text passages (500 passages each for human and Al-generated text classes) of 200 tokens in length. XSum contains long news articles in its "document" feature. To evaluate the
16

---

Published in Transactions on Machine Learning Research (01/2025)
robustness of our attacks to distribution shifts, we include more datasets. We use PubMedQA, which is medical text dataset. Kafkai dataset (Pu et al., 2023) contains real and fake articles (generated using privately fine-tuned OpenAl models) from 10 different domains, such as cybersecurity, SEO, and marketing. It is generated using Kafkai text generation service (Kafkai, 2020).
A.1 Additional Experiments with Llama-2-13B
In this section, we provide evasion attack results with Llama-2-13B as the target model. We use a smaller Llama-2-7B model for recursive paraphrasing. In this section, we consider the XSum dataset for the experiments. As shown in Table 4, the detection performance (TPR@1%FPR) of detectors degrade with recursive paraphrasing.
likelihood threshold
log. rank threshold
perturbation 10 z
entropy threshold
pertarbation 10
Kuditipudi et al.
perturbation 1
rank threshold
perturbation
roberta- bosse
Longformer
roberta-
MAGE
Detector
No attack
0.32
0.32
0.612
0.048
0.992
0.148
0.98
0.0
0.457
0.772
0.672
0.648
0.988
ppl
0.1
0.1
0.204
0.0
0.652
0.1
0.564
0.104
0.405
0.476
0.404
0.6
0.532
pp2
0.142
0.142
0.124
0.02
0.536
0.076
0.444
0.116
0.4
0.454
0.316
0.556
0.356
pp3
0.04
0.04
0.068
0.036
0.516
0.076
0.432
0.104
0.421
0.424
0.304
0.56
0.322
pp4
0.04
0.04
0.052
0.02
0.492
0.08
0.412
0.12
0.427
0.436
0.296
0.504
0.292
pp5
0.068
0.068
0.06
0.0
0.476
0.08
0.388
0.12
0.421
0.432
0.304
0.524
0.264
Table 4: Detector performance of various detectors where Llama-2-13B the target model. The attacker uses smaller Llama-2-7B model for recursive paraphrasing. The first four detectors with prefix "perturbation" are DetectGPT (Mitchell et al., 2023) variants. The next four detectors with sulfix "threshold" are threshold-based zero-shot detectors. The next four detectors are trained detectors (Li et al., 2024; OpenAl, 2019). Kuditipudi et al. (2024) is a distortion-free watermarking technique.
A.2 Watermark-baat Detectors
In this section, we analyze the soft watermarking scheme in Kirchenbauer et al. (2023a). We use the powerful DIPPER paraphraser from Krishna et al. (2023) with 11B parameters for recursive paraphrasing attack on the watermarking detector. On average, five rounds of our recursive paraphrase attack take around 36 seconds per text passage, 300 tokens in length. OPT-13B is used to measure the perplexity scores for all the settings. As shown in Table and Appendix B.1, we perform a human study over the XSum datasct to evaluate the semantic drifts in our recursive paraphrasing framework. The MTurk human evaluation reveals that 70% of our recursive paraphrases maintain quality content preservation, and 89% of our recursive paraphrases have high-quality text or grammar.
Figure 11 shows the performance of the soft watermarking detector in multiple settings. In all the settings, the detection performance drops as rounds of recursive paraphrasing proceed with slight degradation in perplexity scores. After two rounds of paraphrasing (pp2), the detection performance (TPR&1%FPR) in all the settings drops below 50%. Best of ppi, which selects the paraphrase with the worst detection score, significantly degrades the detection performance to below 10% all the settings with degradation of 1.5, 0.5, 2.0, and 2.7 in perplexity measures.
A.3 Zero-shot and Trained Detectors
In this section, we analyze the zero-shot and trained detectors in prior litcrature (Mitchell al., 2023; Solaiman al., 2019; Gehrmann et al., 2019; Ippolito et al., 2019; OpenAI, 2019). We use the T5-based paraphraser (Damodaran, 2021), Parrot, to paraphrase the AT-generated text and use OPT-13B to measure
17

---

Published in Transactions on Machine Learning Research (01/2025)
the perplexity scores for all the settings. We perform our experiments the XSum (Narayan et al., 2018); PubMedQA (Jin et al., 2019), and Kafkai (Kafkai, 2020) datasets with GPT-2-Medium and OPT-1.33 as the target generative models. In Figure 12 (ROC curves) and Tables 5 (TPR@17PPH values) and 6 (AUROC scores), we present our results. 1d, 1z, 10d, and 10z in Tables 5 and 6 refer to different variants of the DetectGPT (Mitchell et al., 2023).
Figure 12 shows the performance of various zero-shot and trained detectors in multiple settings. The performance of these detectors drops significantly when the Al-generated text is paraphrased, and when given 5 queries to the detector, an adversary can fool most detectors effectively. Some detectors like OpenAl's RoBERTa-basod models are more resilient on datasets like XSum, but are not reliable on other datascts like Kafkai. The perplexity scores of the GPT-2 generated text before any paraphrasing were 15.58 for XSum, 12.80 for PubMedQA, 19.11 for Kafkai, while the perplexity of OPT-1.3B generated text was 9.31. After paraphrasing, the perplexity scores are 20.06, 16.45, 20.01, and 13.96, respectively.
A.4 Retrieval-based Detectors
In this section, analyze the retrieval-based detector proposed in Krishna et al. (2023). We show that our recursive paraphrasing attack is effective in breaking their detector. We use the 11B parameter DIPPER paraphraser (Krishna et al., 2023) for our attack. OPT-13B is used to measure the perplexity scores in all the scttings.
Figure 13 shows the performance of the relrieval-based detector in multiple settings. In all the settings, the detection accuracy drops as rounds of recursive paraphrasing proceed with only a slight degradation in perplexity scores. We observe that the detector works well after single round of paraphrasing (pp1). However, after five rounds of paraphrasing, Best of ppi reduces the detector's accuracy to close to 50% with only a slight degradation in perplexity scores. We also find that we can easily spoof the retrieval-basee detector as discussed in $3 to deteriorate the detector's performance to 0%. Note that retrieval- based detectors are concerning since they might lead to serious privacy issues from storing users' LLM conversations.
B More Details on AI Paraphrasers
B.1 Human Evaluation Study on Paraphrases
Apart from measuring the perplexity scores of the paraphrases using OPT-13B and performance the SQUaD-v2 benchmark, perform two human evaluation studies to investigate the quality of the paraphrases from DIPPER and passages and their corresponding five rounds of recursive paraphrases (pp1 to pp5) for cach of the paraphrasors for human evaluation. Each paraphrase is evaluated by 3 unique MTurk workers. We use the same setup as Krishna et al. (2023) for our human study. As shown in Figure 14, users are given a source text with some highlighted portion. The non-highin non-highlighhtd portion of the source text is input into the target OPT-13B model that generates watermarked text which is highlighted for the user's reference. DIPPER or LLaMA-2 paraphrases of the highlighted text are provided as the paraphrasing. The user is supposed to evaluate the quality of the paraphrases with respect to the highlighted watermarked text. They are supposed to rate it on a Likert scale of 1 to 5. See Tables 7,8 for the evaluation summary on content preservation of DIPPER paraphrasers based on the user study. Tables 9,10 show the summary of the evaluation of text quality/grammnn /grammarr of the paraphrasos. For the content preservation study, we use the following Likert scale: preserves the meaning of the source but differs in words and/or structure. 4 preserves most information in the source but differs in some minor factual details. 3 reserves some information in the source but differs in certain significant ways. 2 - topically related to the source but most information in the source is not preserved. - not topically related. For the text quality or grammar quality study, we use the following Likert scale: 5 - the paraphrase has excellent grammar/qualin clear and correct with minor grammatical errors. - the paraphrase has few grammatical errors, but remains clear and comparable to highlighted source text. the paraphrase has significant number of grammatical errors, but remains understandable. the paraphrase is inferior to the highlighted source text with a lot of grammatical errors, may be difficult. to comprehend.
we use for the paraphrasing attack. We pick 20 random watermarked
with respect to the highlighted source. 4 the paraphrase is
18

---

Published in Transactions on Machine Learning Research (01/2025)
DetectGPT
Threshold by
RoBERTa
1z
10 d
10
Likeli- hood
Rank
Log Rank
Entropy
Base
Large
OPT-1.3B XSum
No attack
0.079
0.079
0.083
0.125
0.237
0.382
0.288
0.017
0.694
0.956
pp attack
0.014
0.014
0.018
0.006
0.006
0.006
0.006
0.326
0.025
0.479
5 pp attack
0.0
0.0
0.005
0.0
0.004
0.001
0.002
0.202
0.003
0.244
GPT-2 on PubMedQA
No attack
0.05
0.05
0.598
0.481
0.085
0.379
0.144
0.029
0.748
0.902
pp attack
0.052
0.052
0.19
0.054
0.015
0.042
0.017
0.202
0.181
0.606
5 pp attack
0.0
0.0
0.031
0.002
0.008
0.01
0.012
0.135
0.088
0.452
GPT-2 on Kafkai
No attack
0.077
0.077
0.669
0.625
0.088
0.352
0.085
0.0
0.048
0.006
pp attack
0.056
0.056
0.125
0.081
0.004
0.021
0.004
0.002
0.01
0.0
5 pp attack
0.0
0.0
0.023
0.006
0.002
0.004
0.002
0.0
0.0
0.0
GPT-2 on XSum
No attack
0.169
0.169
0.599
0.326
0.114
0.444
0.186
0.026
0.881
1.0
pp attack
0.038
0.038
0.084
0.015
0.003
0.005
0.003
0.411
0.105
0.925
10 pp attack
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.058
0.02
0.792
Table 5: TPR@1%FPR for trained and zero-shot detectors in different settings. For all attacks, we use the T5-based paraphraser. Here, "pp attack" refers to the paraphrasing attack where the AT output is paraphrased by the based model. "i pp attack" refers to the setting where the attacker has black- access to the detector. Here, the paraphraser generates "i" paraphrases for every passage, and the attacker selects the passage that has the worst detection score after "i" queries to the detector.
DetectGPT 1z
Threshold by
RoBERTa
d
10
10
Likeli- hood
Rank Log Entropy Rank
Base
Large
OPT-1.3B on XSum
No attack
0.769
0.769
0.9
0.859
0.918
0.844
0.943
0.482
0.974
0.998
pp attack
0.487
0.487
0.453
0.41
0.241
0.387
0.282
0.868
0.562
0.945
5 pp attack
0.162
0.162
0.244
0.182
0.153
0.216
0.181
0.821
0.316
0.9
GPT-2 on PubMedQA
No attack
0.816
0.816
0.973
0.955
0.804
0.796
0.892
0.615
0.982
0.998
pp attack
0.671
0.671
0.796
0.743
0.4
0.497
0.494
0.798
0.823
0.98
5 pp attack
0.33
0.33
0.601
0.541
0.327
0.314
0.409
0.752
0.712
0.967
GPT-2 on Kafkai
No attack
0.814
0.814
0.976
0.971
0.865
0.86
0.89
0.394
0.817
0.86
pp attack
0.661
0.661
0.757
0.742
0.497
0.719
0.515
0.651
0.486
0.629
5 pp attack
0.353
0.353
0.532
0.513
0.412
0.627
0.426
0.576
0.358
0.53
GPT-2 on XSum
No attack
0.837
0.837
0.976
0.949
0.879
0.868
0.93
0.617
0.993
1.0
pp attack
0.566
0.566
0.587
0.524
0.171
0.277
0.23
0.916
0.726
0.995
10 pp attack
0.115
0.115
0.202
0.177
0.075
0.104
0.108
0.744
0.464
0.983
Table 6: AUROC for trained and zero-shot detectors in different settings. Here, "pp attack" refers to the paraphrasing attack where the AI output is paraphrased by the T5-based model. "i pp attack" refers to the setting where the attacker has black-box access to the detector. Here, the paraphraser generates "i" paraphrases for every passage, and the attacker selects the passage that has the worst detection score after "i" queries to the detector.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Based the evaluations, 70% and 83% of the paraphrases rated high quality in terms of content preservation for DIPPER and LLaMA-2, respectively. 89% and 88% of the paraphrases are rated to have high-quality text/grammar for DIPPER and LLaMA-2, respectively. Our human study shows the tradeoff of our recursive paraphrase attack strength with text quality for the watermark- based detector.
B.2 LLaMA-2 Chat Template
Below, we provide the chat template we use to employ LLaMA-2-7B-Char as paraphraser.
You paraphraser. You given input passage 'INPUT: You should paraphrase 'INPUT" to print 'OUTPUT: 'OUTPUT" shoud be diverse and different much as possible from 'INPUT" and should not copy any part verbatim from 'NPPTT. 'OUTPUT" should preserve the meaning and content of 'INPU' while maintaining text quality and grammar. 'OUTPUT' should not be much longer than 'IPPUT You should print 'OUTPUT' and nothing else so that its easy for me to parse. User Prompt: INPUT: [Add input passage here]
System Prompt:
Below, we provide the chat template we use to employ LLaMA-2-13B-Chal as a question-aneswering model.
System Prompt: You are given context OC: [Add context passage herel' and question 'Q: [Add question here]: Let 'A' be the answer to question 'Q' solely based context 'C'. You are given the true answer AA1 [Add ground truth answer here]" for question 'Q'
User Prompt: INPUT: Do answers 'Al' and 'A' match? You SHOULD only be printing either 'YES' or 'NO:
20

---

Published in Transactions on Machine Learning Research (01/2025)
1.0
1.0
0.8
0.8
0.6
0.6
TPR
TPR
0.4
AUROCI0,
0.4
Weremmrding AUROCIO 999,
SPPRO 995,
9p1
977, TPRG1SIFRC 807,
PP1
862
AUROC:0:55 9144, TFRS)1NFFRD ppl.7.8
892
AURDC:O PAGIENIPI ppts.3
0.2
AUROC:0 853, TPA.91PPR::. 309, ppir8.5
0.2
993
AUROC TPRg15IPPR 244,
F019.0
AURDC:0 922, /PRgINFFR:0-2 ppt:7.
AUROCIO 808.
TPSg1S/PR:: ppl30.3
AUROC:0.763. 119ggS5FFR..115
795, TPR@ANHPR:O
Bost pp Random
AUROC:0.672. TRRg)38FFR:D0077
077.
pa
pp:11.5
0.0
AUROC:0.58 5500 TFRg15/FR000000
0.0
Random
AUROC:O
0.8
1.0
0.4
0.6
0.8
1.0
FPR
FPR
(a) XSum with OPT-13B paraphrased with DIPPER
(b) XSum with OPT-13B paraphrased with LLaMA-2
1.0
1.0
0.8
TPR
TPR
0.4
Vetermarking
pp17.6
Watermanking AUROC:O 999. 1PR:LNNPPR por15.
ppt
AUROC:O TPR.1150PP
sp:8
Pp1
AUROC:0.975, TPR:1NTFR:: pp:15.2
AUROC:0 017, 19R 9999
ppt.5.7
EP2
AUROC-B. 8888, 1PP001511PR:04744
ppc:5.22
0.2
AUROC:0 840, TRRg1195PR, 0.232, 9p1..7
0.2
663
AUROC:0.815, TPR@)1/FP::0.22 295.PRR.0225,
Pp016.0
AUROC:0 TPR@115FPR
pp1.3.5
AUROC:0.761. TFR:)N/FR:O pp:15.9
ppt:9.8
PP5
ppi
AURDCCJ
spt/3.1
AUnOC:00
pok:6..4
0.0
..... Random
AUROC:0 500. 199:1155PP.00000
0.0
.... Random
AUROC.0.500
TPRg)15#FR::0000
0.2
1.0
FPR
FPR
(c) XSum with OPT-1.3B paraphrased with DIPPER
(d) XSum with M-d--um paraphrased with DIPPER
1.0
1.0
0.8
0.8
0.6
0.6
TPR
TPR
0.4
Wateraaring AUROC- 1,000 TPROISFPRO 998, ppl6.1
0.4
Watermrriig AUROC: 1.000 TRRG15/9R000000 9p16
AUROC:0.000 045, TRRg195AAL 0.656, sp17.4
AUROC:0.977. TPSgLSFPPO.
AUROC:0 TRRG11SFPR
AUROC:O.905 TPA@11FFR
0.2
pp17.
0.2
PD3
RP4
ppt.5.0
PP4
AURO0:0.890 1PNg11N/PR-.736
D03
AUROC:00.6 683,
(Fxg115F99.011244
ppt.3.3
P05
AUROC:0.774, TPA@11%FP..0.175
po:..4
Best of ppl
AUROC.0.550 TPR:g11SFPR:O 0044
spl.3.1
Best of pp!
AUROC: TRR:@11-PP..000 0.2
1pt:9
Random
AUROC:0
AUROC:0.500 000...00000000 TPR@11//PR0000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
FPR
(e) PubMedQA with OPT-1.3B paraphrased with DIPPER
(f) Kafkai with OPT-1.3B paraphrased with DIPPER
Figure 11: ROC plots for soft watermarking (Kirchenbaner et al., 2023a) with our recursive paraphrasing attacks. AUROC, TPR@1 "PR@1%FPR, AFPR, and perplexity scores measured using OPT-13B given in the legend. Detection performance the XSum dataset using 3 different LLMs OPT-13B, OPT-1.3B, and GPT- 2-Medium
evaluated in (a), (c), and (d), respectively. (b) show the performance of the detector with recursives paraphrases from LLaMA-2-7B. Chat. (c) and (f), respectively, show the performance of the detector on two datasets - PubMedQA and Kalkai with distribution shifts using OPT-1.3B. In all the settings, we observe that the detection performance of the watermarking based detector degrades but with a tradcoff in the text perplexity measures after the attack.
21

---

Published in Transactions on Machine Learning Research (01/2025)
1.0
1.0
Thue Positive Rate
Positive Rate
True Positive Rate
0.6
0.6
0.6
0.2
0.2
0.0
0.0
0.0
False Positive Rate
False Positive Rate
False Positive Rate
perturbation_11 __d
perturbation, 10
likelihood threshold Mielihood_theesholdd
log rank threshold
roberta base openai detector
openai-detector
perturbation 1
perturbation 10
rank threshold
entropy_threshole threshold
roberta-lnge-opeenaldeettet -eeteccor
(a) Kafkai with GPT-2
1.0
1.0
1.0
True Positive Rate
True Positive Rate
True Positive Rate
0.6
0.6
0.6
0.4
0.2
0.0
0.0
0.0
10
10
10
False Positive Rate
False Positive Rate
False Positive Rate
perturbation __d
perturbation 10_d
likelihood threshold teeiihood_threshhldd
log sog_rank_eee threshlld
roberta base base-openai openai-detector detector
perturbation 1,z
perturbation 10
rank threshold
entropy threshold
roberta-arge-gpenal-eeetet -eteccoo
(b) PubMedQA with GPT-2
1.0
1.0
1.0
True Positive Rate
True Positive Rate
irue Positive Ratte
0.6
0.6
0.6
0.4
0.4
0.0
0.0
0.0
10-3
10
10-1
10
10-1
10
10-2
10-1
10
False Positive Rate
False Positive Rate
False Positive Rate
perturbation 1_d
perturbation _0_d
likellhood threshold theesholdthreshhlld
log_rank thrksthre
roberta-base-opeaii-dee base-opemaiddetector
perturbation 1,z
perturbation 10
rank threshold
entropy threshold
roberte-lnge--peaaaddetctoo
(c)
XSum with GPT-2
1.0
1.0
1.0
True Positive Rate
True Positive Rate
True Positive Rate
0.8
0.6
0.6
0.4
0.4
0.4
0.0
0.0
0.0
10-3
10-1 False Positive Rate
10-2
100
10-3
10-2
10-1
10-3
10-2
10-1
False Positive Rate
False Positive Rate
perturbation 1
perturbation_ ____d
likellhood threshold likeshlood_hrreshlld
log hog_rank_oog threshld
base-openai-detector
perturbation
perturbation 10.
rank threshold
entropy threshold
abbertalargg-ooenai-deee
(d) XSum with OPT-1.3B
Figure 12: ROC curves for performance of various zero-shot and trained detectors for different models and datasets (Left) before attack, (Middle) after paraphrasing attack, (Right) applying paraphrasing attack with multiple queries to detector.
22

---

Published in Transactions on Machine Learning Research (01/2025)
7.6
15.9
Detection performance (%6)
100
8.3
8.7
Detection performance (%)
100
15.2
B.7
15.2
80
8.9
80
16.0
15.9
60
60
16.6
16.5
40
40
Retrieval-basee detector
20
Netrieval-baseer detector
Random detector
Random
8.3
15.2
No attack
pp1
pp2
pp4
pp5
Best Spoo-
No attack
pp1
pp2
pp3
pp4
pp5
Best Spoo- ppi
ppi
fed
fed
(a) XSun with OPT-133
(b) XSum with GPT-2-Medium
6.1
7.4
100
6.4
8.0
Detection performance (%6
7.8
Detection performance (%)
8.5
7.9
8.7
80
8.0
80
B.9
8.3
8.2
9.4
9.1
60
60
40
40
20
Retrieval-basee detector
20
Retrieval-bassee detector
Random detector
Random detector
7.4
8.0
No
ppl
pp2
pp3
pp4
pp5
Best
Spoo- fed
No attack
ppl
pp2
pp3
pp4
pp5
Best Spoo- ppi
attack
ppi
fed
(c) PubMedQA with OPT-1.3B
(d) Kafkai with OPT-1-3B
Figure 13: ROC plots for soft watermarking (Kirchenbauer et al., 2023a) with our recursive paraphrasing attacks. AUROC, (PR@I%FPR, and perplexity scores measured using OPT-13B are given in the legend. Detection performance on the XSum dataset using two different LLMs OPT-1.3B and GPT-2-Medium are evaluated (a) and (b), respectively. (c) and (d), respectively, show the performance the detector on two datasets PubMedQA and Kafkai with distribution shifts using OPT-1.3B. In all the settings, we observe that the detection performance of the watermarking- based detector reduces with tradeoff in the perplexity measures of text after the attack.
23

---

Published in Transactions on Machine Learning Research (01/2025)
View instructions
Read detailed instructions carefully before proceeding (click on "View Instructions" above).
Given source text:
Forest Green, promoted from the National League, will host MK Dons in their first appearance in the competition, while FA Cup giant-killers Lincoln will be away to Rothernm.. The 35 ties will be played in the week commencing Monday, August. Hull City and Middllebbrough have been handed bye into the second round, having finished above Sunderland in the Premier League last season. There was confusion after the draw, which was streamed live from Bangkok, where the competition's new sponsors, energy drink company Carabao, are based. A list of fixtures displayed on the stream showed Charlton drawn against two clubs, while AFC Wimbledon were also wrongly recorded as being at home to Swindon the Dons were drawn at home to Brentford, and Swindon will be away Norwich. And Forest Green were listed as being away Wolves, who were in fact drawn at home to Yeovil. The live stream was also hampered by sound problems, with listeners on some clubs' websites unable to hear the draw. The draw was conducted by former Premier League referee Mark Clattenburg, who had been involved in the draw for the first match of the season. He is employed by Premier League broadcast partner Channel as television match official. The draw for the first round was conducted on live television at 10.30pm Thai time, with the match to shown on Thailand's Channel 5 as part of the BBC's coverage of the tournament. The fourth round was drawn on live television at 10pm Thai time, with matches to be shown on Thailand's Channel 5 as part of the BBC's coverage of the tournament. The draw for the fifth round was conducted on live television at 10. 10.25pm Thai time, with matches be shown on Thailand's Channel drawn on live television at 10. 10.15pm Thai time, with matches to be shown on Thailand' s Channel 5 as part of the BBC's coverage of the tournament.. The seventh round was drawn on live television at 10.15pm Thai time, with matches to be shown on Thailand's Channel 5 part of the BBC's coverage of the tournament.. The draw for the eighth round was conducted on live television 10m Thai time, with matches be shown on Thailand's Channel as part the BBC's coverage of the tournament. The draw for the ninth round was conducted on live television at 10.15pm Thai time, with matches to be
part of the BBC's coverage of the tournament. The sixth round was
Paraphrase of highlighted text:
Mark Clattenburg, a former Premier League referee, who officiated the opening game of the season, was in charge the draw. He is now employed by the FA's broadcast partner, Channel 5, as a television match official. The draw for the first round was made on live TV at 10. 10.30pm local time, with Channel 5 in Thailand broadcasting the event part of the BBC's coverage of tournament. The fourth round was drawn live on Thai TV 10pm local time, with matches be screened on Channel in Thailand as part of the BBC's coverage of the competition. The draw for the fifth round took place live on Thai TV at 10.25pm local time, with Channel 5 in Thailand to broadcast the ties as part of the BBC's coverage. The sixth round was drawn live on Thai TV at 10. 15pm. Channel 5 in Thailand to show matches part of BBC's coverage. The draw for the eighth round was made live on Thai TV 10. 10.15pm. Channel 5 Thailand to show matches as part of BBC's coverage of the competition.
Which of the following best describes the quality of the paraphrase?
O 5 Approximately equivaient: the paraphrase preserves the meaning of the source but differs in words and/or structure
O 4 Nearly equivalent the paraphrase preserves most information in the source but differs in some minor factual details
O 3 Somewhat equivalent: the paraphrase preserves some information in the source but differs in certain significant ways
O 2 Topically related: the paraphrase is topically related to the source but most information in the source is not preserved
O 1 Not topically related: the paraphrase is not topically related to the source and preserves no information
Write a short reasoning for your answer:
Write here...
Submit
Figure 14: MTurk user interface for human evaluation of paraphrases for content preservation.
24

---

Published in Transactions on Machine Learning Research (01/2025)
Average rating
Sum of 5 (%)
Approx. equivalent (%)
nearly equivalent (%)
Somewhat equivalent (%)
2 Topically related (%)
Topically unrelated (%)
ppi
i=1
4.0 0.8
70.2
29.8
40.4
29.8
0.0
0.0
i=2
0.8
77.2
33.3
43.9
19.3
3.5
3-3
3.9 0.9
63.2
33.3
29.8
33.3
3.5
0.0
1-4
4.2 = 0.9
49.1
30.9
14.5
5.5
1=5
3.7 1.1
61.4
29.8
31.6
21.1
17.5
0.0
ppi
4.0
70.4
35.1
35.3
23.6
6.0
Table 7: MTurk human evaluation of recursive paraphrases with DIPPER for content preservation. ppi represents the ith round of recursive paraphrasing.
Average rating
Sum of
Approx. equivalent (%)
nearly equivalent (%)
Somewhat equivalent (%)
Topically related (%)
1 Topically unrelated (%)
ppi
5 & (%)
1=1
4.37 0.63
45.0
46.67
8.3
1=2
4.18 + 0.67
85.0
33.33
51.67
15.0
0.0
0.0
i=3
3.93 # 0.71
80.0
21.67
58.33
16.67
3.3
0.0
i=4
3.85
0.78
78.33
21.67
56.67
16.67
5.0
0.0
i=5
3.7
80.0
18.33
61.67
11.67
8.33
0.0
ppi
Table MTurk kuman evaluation of recursive paraphrases with LLMMA-2-BB-Chal for content preservation. ppi represents the 1th round of recursive paraphrasing.
Average rating
Sum of
5 Excellent
4 Good
Fair
2
Adequate (%)
1 Poor
ppi
5 4 (%)
(%)
(%)
(%)
(%)
i=1
4.28:t 0.67
87.72
40.35
47.37
12.28
0.00
0.00
1=2
4.12 + 0.50
92.98
19.30
73.68
7.02
0.00
0.00
1=3
4.12 0.53
91.23
21.05
70.18
8.77
0.00
0.00
1=4
4.11 4:064
84.21
26.32
57.89
15.79
0.00
0.00
1=5
4.07 + 0.53
89.17
17.54
71.93
10.53
0.00
0.00
All ppi
4.14 0.58
89.12
24.91
64.21
10.88
0.0
0.0
Table 9:
Human evaluation of recursive paraphrases using MTurk for text the th round recursive paraphrasing.
ppi represents
Average rating
Sum of
5
Excellent (%)
Good
Fair (%)
Adequate (%)
1 Poor
ppi
5 4 (%)
(%)
(%)
i=1
4.62:t 0.55
96.67
65.0
31.67
0.00
0.00
2
4.28 + 0.73
83.33
45.0
38.33
16.67
0.00
0.00
1=3
4.26
0.65
88.33
43.33
45.0
11.67
0.00
0.00
1=4
4.22 4:0664
88.33
38.33
50.0
11.67
0.00
0.00
1=5
4.17 + 0.74
83.33
40.0
43.33
15.0
1.67
0.00
All ppi
4.32
0.35
88.0
46.33
41.67
11.67
0.33
0.0
Table 10:
Human evaluation recursive paraphrases using MTurr for text quality coallty/grammet /grammar. ppi represents the 1 th round of recursive paraphrasing.
25

---

Published in Transactions on Machine Learning Research (01/2025)
B.3
Example Paraphrases
Description
Text
The draw was conducted by former Premier Leugue reforve Mark Clattenburg, who had been invalved the draw the first match of the season. He is employed by Premier League brodrrsst partner Channel as television match official. The drow for the first round was conducted on live television at 10.30pm Thai time, with the match to be shown on Thailand's Channel as part of the BBC's coverage of the tournaennt. The fourth round was drawn on live television at 10pm Thai time, with matches to he shown on Thailand's i Channel as part of the BBC's 's coverage of the tournament. The draw for the fifth round was conducted on live selevision at 10.25pm Thai time, with matches to be shown on Thailand's Channel as part the BBC'ss coverage of the tournament. The sixth round was drawn Thailand's is Channel as part of the BBC's coverage of the touruament. The seventh round was drawn live television at 10.15pm 15pm Thai time, with matches to be shown on Thailand's Channel 5 as part of the HBCCss coverage of the tournament. The draw for the eighth round was conducted on live television at 10.m Thai time, with matches to be shown on Thailand's Channel 5 as part of the BBC's coverage of the lournament. The draw for the ninth round was conducted on live television at 1pm Thai time, with matchrs be
Tupul
live television at 10. 15pm Thai time, with matches to be shown
Mark Clattenburg, a former Premier Lengue referee, who officiated the opening game of the season, was in charge of the draw. He now employed by the FA's broadcast partner, Channel 5, as television match official. The draw for the first round was made on live TV at 10.30pm local time; with Chaumel 5 in Thailand broadcasting the event as part of the BBC's coverage of the tourmament The fourth round was drawu live un Thai TV at 10pm local lime, with matches to be screened on Channel 5 in Thailand as part of the BDC's coverage of the competition. The draw for the fifth round took place live on Thai TV at 10 10,25pm local time, with Channel 5 in Thailaud to broadcast the ties as part of the BBC's coverage. The sixth round was drawn live on Thai TV at 10.15pm. Channel made live on Thai TV at 10.15pm. Channel 5 in Thailand to show matches as part of BDC's coverage the competition.
Paraphraee
Thailand show matches as part of BBC's cuverage. The draw for the eighth round was
"We had to cancel his operation and put off on him, he was supposed to have it on Christmas Eve." He said Josh and his family had boen in touch with some people from the US and Austealia who are in Belfast right now. He is one of the first people from the US to come to Belfast to meet and speak with Josh, and the rest of the family will visit from the US in the next few days. 'Josh has become worldwide man's man, and has been incredible to how everybody has taken him. He's aubassadr for Irish people and for Ireland, and the people are really behind him. He's great aubasador for Ireland and for his country and it has been really good." ' Josh was taken to Queen's University Hospital for surgery and they discovered and cut down his appendix, which had been causing him pain for more than a year. The surgery took place on January 11. He is still on painkillers, and the family are eucouraging people to keep praying for his recovery. The hashtag # # forjofo and the post and the meesage messages are now being shared widely throngh social media by people from the US and Australia. The most recent tweets from his father's account are about his recovery in Belfast. He also being accompenied by "pray warriors' team and the hashing #prayforjosh is now being used by other campaigners to urge people to send support to Josh. The church is also asking people to help
Inpul
"His operation was cancelled and put back - it was due to be on Christmas Eve." He said that the family and friends of Josh have been in touch with people from the United States and Australia who are currently in Belfast, He himself was one of the first to arrive in the city to meet and talk to the temageer and more of the family will arrive in a few days. "Jush has become a 'glohall' man and it's been amazing to see how people have taken to him. He's real Irishman and an ambesedor for Ireland and it's been really good for him." He said that
Jamuary 11th, when he taken Queen's University Hospital in Belfast for the operation, large primary cancer and five smaller growthss praying for his rocovrry. The post with the bashing forjosh and the messages it contains have now bren shared all over social media by people in the US and Australia. The last few posts on his father's 's Twitter account are about his recovery in Belfast. He has also been surrounded by a team of prayer warriors and the campaign has now widened, with people using the tag #ppryforjoeh to urge hers show their support for him. The church is also calling on people help in other ways.
Parapluae
remsved.. He is still recovering and the family usk people to keep
I've enjoyed my time with the tram.' Wales, who won 11 medals in total in Brazil, failed to make the podium at three consecutivn Games from 1996 to 2008 before Doull stepped into the team and helped to end the long wail for a Welsh team pursuit gold. The Websh team were in fine form on Saturday as they captared their 11th medal, equalling the reordd of 14 won in 1972. The Welshh tram'ss is rum to gold this year was docunntedd. Geraint Thomas, the 2008 and 2012 Olympic champion, set an Olympic record of mimutes 59.4 secouds
help Great Britain to gold. Born Cardiff but living in Aberystwyth, Authony Iwan Thomss was selected one of the two Welsh teack riders for Rio and carned a silver medal in the men's is road race, which he won in August. "It's a dream rome true to come bark and win Olympic gold here," " said Thomas. "To do it in this stadium, it's just surreal. I can't 't explain it. Let me get home and I'll be really grateful to everyone." ! He missed the 2012 Games through injury but returned to gold winning form in 2016 when he won his third consentivve world championmliip in the men's road race, finishing second in Rio. "It's real honour to be on the podium today,' added the 35 year-ol. "There's lot of times when you think 'this is it' and you think
Tupul
I've loved every minute with the team. " Wales, who won 11 medals Brazil, had not finished the podium at three consecutive Olympic games from 1996 to 2008 before Doull helped end the country's s long wait for team pussit gold medal. The nation's 11th medal on the last day equalled the record of 14 set in 1972. So many stories of suecess for the Welsh riders in Rio have already been written. Born in Cardiff but now based in the Ceredigion resort of Aberystwyth, the son of Geraint Thomas won silver in the men's road race. Thomas said: 'It's just a dream come true to come back and win gold here and it's even more special do in this stadium. If get home I'll thank everybody. 'hhe 36- year-old had to miss the 2012 Olympic games with injury but was back to winning ways in 2016 when he retained his world title and finished second in Rio. "There's so many times when you think, "This is the day' and it never comes.'
Paraphrase
Examples of paraphrased passage from the XSum dataset. The paraphrasing is performed using Table 11:
DIPPER (Krishna et al., 2023).
26

---

Published in Transactions on Machine Learning Research (01/2025)
Description
Text
The foree has said it has intention of changing policy use of force, and will continue to the weapors part of its role policcng the public. A spokesnaa for Scutland Yard suid: "Offivers in Soathaud will contimue to have a range of weagions on on-hand, including tasers, pepper spray and a handgun and they will be used in accordance with Taser International guidelines, "We also have well established policies governing the use these weapous and the use of farce, and the use of force judged proportionate when appropriate." The move was welcomed by a leading rights campaign group. John O'Donnell, the head of the High Court, said Tisees should have been compatsory in Seotland in 2015 following the death of a journalist in custody in Newcastle and the deaths of two more while police custody. He suid: "The use of these weaponss by police Soutland lus been widely condemned and the High Court ruled they should be used to proteet the public.' A spokesman for the Deportment for Transport said: "No police force should decide for itself what constitutes uecessary force and the use of these weapons by police Scotland has been widely condemned. "The use of these weapons by officers must be subject to strict rules which safeguaed the public and the officess' rights. e The London police suid they 'do not comment intelligence matters'. .?
Topat
The Met said it had no plans to change Its polley on the use of force and said the Tasets would be used "when it is appropriste and proportionate" A spokesman suid: "Ofllcers in Scotland will continue to earry a range of weapors including Tasers, pepper spray and bandguns. "We have well-established policies on use these wespons and the use of force in general and we will continue rnsure these are followed." It was weleonee by the Scottish Human Rights Commission. The high court judge, Lord O' O'Doueell, said in February that the use of Thaus by police in Scotland should have been compubory in the wake of the death of a jourmalist in police custody in Neweastle, and the subsequent deaths of two more people in police custody, He suid: "The use of these wespous by police Scosland has been widely condenned and the high court has ruled they must be used to protect the public." The Department for Transport said police should not be able to decide for themselves when it was appropriste to use forer. "The use of these wespons by officers must be subject to strict rules which protect the public and officers' rights. The Met said it did not comment on intelligence matters. Would you like book an appointment with the British Transport Police!
Paraphrase
"He gets on with me. have uo doubt about his motives." ! The move is also an enormoou gamble but Konkush, a self-madee millionaire who made his fortune as a real estate mogul in Dubai, is determined to make sure the gamble pays off. He is willing, he says, to let the man who guided Salford seven Grand Final appearances head great project. 'He has incredible track revord of creating people and businessess" suid Noble. 'He has dome it in Qutar, in Dubai, in America, here." He is also one of the gume's most ruthless businessmmn, Konkush, who moved to the UK from Sudan with his parents as a boy and has spent, the post years building his empire of clubs and busineses, las seen the game of rugby lenguee decline dramatically over the past years. He claims that the game has never been more popular than it is today. He la also a fervent supporter of the game and this makes his interest in rughy league rven more compelling. Salford have been in dire stritt and Konkash, who has ploughed much own money in the club, has promised to help them become one the great clubs of the game. The club has all the right qualities and Konkush wants make sure it happens. He has spent the past two weeks scouting for players to recruit and has already seen the arrival of two promising youngsters. The Reds need players to make them competitive
Input
He addod: 'I know him. I have no doubt about his intentions." It's a huge gumble for the self-made millionaire who has made his money in property in Dubai He is happy to eutrust the man who took Salford to seven Challeage Cup findls with his great plan. He's got an incredible record of turning around people and businesses' 2 Noble said. "He's done it in Qatar, in Dubai, in America, here." He's also one of the most ruthless busiuessmenn in the game. Having arrived in Britain as a boy from Sudan with his parents, he's spent the past 30 years building an empire clubs and businesses. He's also seen the game decline in popularity
Pamphrase
the past decade, but insists it's more papular than ever, He huge supporter of the gume and that's 's why be's is interested in the sport. Salford have been in crisis and he's is promised to help them become one of the great clubs. The club has all the attributus and Konkash is determiued to make it huppen. He has speat the past fortaight looking at players and has already recruited two, He has to make the Reds a more competitive side and has alread brought couple of players who have impressed.
She suid be would not stop attucking and aslaod for help, She said she wanted "peare' but "not deathi. Henderson- McCarroll, of St Nicholas Drive, Newry, admitted mauslaughter while in charge of dangerous drug. She said her actions were "blip in my mind' us result of "bad derision' to take drugs. Justice Trency suid be would not impose a custodial sentenoe on Headerson-Mcc McCrrrll,,,, but instead sentenced her to three years' imptisoument. The judge said he would not impose the maximum sentence for manslonghter given the circumstances, but felt would impose the minimum of and a half years. He told Henderson-MM CaaessoooMMCreeee McCarroll: "He (Mr Girvan) would not be in his right mind if be would not have let his guard down. If there was one thing the jury should have heard- it was that your actions were a blip in my mind. You didn't 's intend to kill him. You were acting self-deencee You poled him and your actions were a blip and a maximum sentence for manslaughter given Henderson's 's previous convictions would have amounted to between five and seven years. The judge said it was "not an uncommon crime' to kill someone in self-defence. He said sentencing Henderson was an 'ugly case of drug-indueed madness. He added: 'He (Mr Girvan) must have suffered terribly."
Input
a lapse in judgmeent." The judge said the
She said he had not stopped attacking her and she called for help She said she wanted "peace" but death. Heuderson- McCarroll, of St Nicholas Drive in Newry, admitted manslaughter while under the influence of a sedative. She sald her actions were the result of a "mistake" after taking drugs. Mr Justire Treay said he would not grant a suspended sentence, but instend would sentence her three years prison. He said be would not impuse the maximum seutence for manssanghter, cireumstanncss but did not foel he should impose the minimum term of two and hulf years. He said to the defendant: "You could not in your right mind have left your guard down, you did not intead to kill him. If there was one thing the jury eught to have heard, it was that your actions were a momentary lapee of reason. You acted in self defence, you poked him with a knife, momentary and lapse reason." The judge said that given the defendant's previous record the maxinum sentence for manslaughter, with minimum term of year, would have been five to seven years. He soid it was not an uncommon crime for somrone to kill in self defenee. But in this case, it was an ugly case of drug induced maducesss He added: "Mr Girsan must have suffered horribly."
the
Paraphaae
your actions
Table 12: Examples of paraphrased passage from the XSum dataset. The paraphrasing is performed using DIPPER (Krishna et al., 2023).
27

---

Published in Transactions on Machine Learning Research (01/2025)
Description
Text
In the year 2014-2015, Prison Link Cymru made 1,099 referrals, and said some offenders waited up to year before finding suitable accommodation. Staff from the charitable organization say that the investment in housing would be cheaper than having to re-lock homeless offenders. The Welsh government said that more people than ever are getting help to deal with their housing problems. Changes to the Welsh Housing Act in 2015 removed priority rights for ex-offenders. Prison Link Cymru, which helps people find accommodation after they have been released, said that women generally had good time, as they had to deal with children and other problems.
Input
In the year 2014-2015, Prison Link Cymru made 1,099 referrals, and said some offenders waited up to year before finding suitable accommodation. Staff from the charitable organization say that the investment in housing would be cheaper than having re-lock homeless offenders. The Welsh government said that more people than ever getting help to deal with their housing problems. Changes to the Welsh Housing Act in 2015 removed priority rights for ex-offenders. Prison Link Cymru, which helps people find accommodation after they have been released, said that women generally had good time, as they had to deal with children and other problems.
pp1
In 2015-16, Prison Link Cymru had 1,099 referrals and said some offenders were living rongh for up to year before finding suitable accommodation. Workers at the charity claim investment housing would be cheaper than locking up homeless repeat offenders. The Welsh Government said more people than ever before were getting help address housing problems. Prison Link Cymru, which helps people find accommodation after they have left prison, said things were generally good for women because issues such as children or domestic violence were often the main reasons they returned to prison.
pp2
fact, it was one of the main reasons why they came back to prison. In the year 2015-16, 1615-6, Prison Link Cymru made 1,099 referrals and said that offenders had lived for up year before finding suitable accommodation. The workers at the charitable organization say that the investment in housing would be cheaper than re-locking homeless offenders. The government Wales says that more people than ever are being helped deal with their housing problems. Prison Link Cymru, which helps people to find accommodation after being released, says that women are generally in good position, as children and other problems are often the main reason why they are returned to prison.
pp3
In the year to the end March, Prison Link Cymru had 1,099 referrals and said offenders had been living rough for up to a year before finding suitable accommodation. Workers at the charity say investment in housing would be cheaper than re-imprisoning homeless repeat offenders. The Welsh Government says more people than ever before are getting help to address their housing problems. Prison Link Cymru, which helps people find accommodation after they have been released from prison, said generally things were good for women because issues such as children and other problems were often the main reasons they returned to prison.
pP4
The government of Wales says that more people than ever before are being helped to deal with their housing problems. the year 2015-16, Prison Link Cymru referred 1,099 people and said that homeless people had lived up to year before finding suitable accommodation. The workers at the charitable organization say that the investment in housing would be cheaper than imprisoning homeless offenders again. Prison Link Cymru, which helps people find accommodation after being released, says that women are generally well, because children and other problems often the main reason why they return prison.
pp5
Table 13: Example of a recursively paraphrased passage from the XSum dataset. The paraphrasing is performed using DIPPER (Krishna et al., 2023). ppi refers to the output after i rounds of recursive paraphrasing.
28

---

Published in Transactions on Machine Learning Research (01/2025)
Description
Text
The final round of fixtures will be played in Belfast on Tuesday, Elsewhere, Ireland's U19 women face Wales on Saturday in their Euro U19 women qualifier. The Irish women are already assured of a third barrd-placed finish and place at next year's finals in Northern Ireland after wins over Bulgaria and Greece. If they win against Wales they will finish second and qualify for play-off for the finals. The Trish U19 women face Wales in Aberystwyth on Suturday. The match kicks off at 1.15mm. The Irish U19 women are in excellent form having won their last. four matches. The witning margins have been 4-0, 4-0, 3-0, 3-0 and 3-0. The Irish U19 women sit fourth in the qualifying group and could qualify for the finals if they beat Wales on Saturday. The Irish women in impressive form having won their last four games and qualified for next year's finals. Saturday's 's game against Wales is one the most important of the campaign as a win will ensure they finish second and qualify for play-off for the finals. The Trish will come up against whoever comes top of the group- Demmark, Portugal, Germany and Sweden with the second place finisher going into a play-off for final spot. The Irish U19 women took down Bulgaria 3-0, won 3-0 away to Grevce and honunered home five against young Sponish team.
Input
Ou Tuesday, the final round of matches will take place in Belfast. Mcsawhile, Ireland's U19 women's 's team faces Wales in their Euro U19 women's qualifter on Saturday in Aberystwyth. The Irish are already assured of a third-placed finish and a spot in next year's finals after their wins over Bulgaria and Greeve. A victory against Wales would see them finish second und qualify for playoff for the finals. The Irish U19 women have been in itopressive form, winning their last four matches by margins of 4-0, 4-0, 3-0, 3-0, and 3-0, They sit fourth in the qualifying group and could qualify for the finals if they beat Wales on Suturday. The Irish team will face the top finisher from the group- Denmark, Portugal, Germany, or Sweden in a playoff for final spot. The Irish U19 women have had strong campaign so far, defeating Bulgaria 3-0, winning 3-0 away to Greece, and scoring five goals against young Spenish team.
pp1
On the eve the final round matches in Belfast, the Irish U19 women's team prepares to face Wales in their Euro U19 women's qualitier on Saturday in Aberystwyth. While Ireland has already secured a third-placed finish and a spot in next year's finals with their wins over Bulgaria and Creece, victory against Wales would see them finish second and qualify for a playoff for the finals. The Irish U19 women have been in impressive form, winning their last four matches by margins of 4-0, 4-0, 3-0, 3-0, and 3-0. They currently sit fourth in the qualifying group and could potentially qualify for the finals if they heat Wales on Saturday. The team will face the top finisher from the group Denmark, Portugal, Germany, or Sweden in a playoff for a final spot. Treland has had a strong campaign so far, defcating Bulgaria 3-0, winning 3-0 away to Greece, and scoring five goals against a young Spanish team.
pp2
On the eve of the final round of matches in Belfast, the Northern Irish U19 women's team prepares to face Wales in their Furo U19 women's qualifier on Saturday in Aberystwyth. While Northern Treland has already secured third-placed finish and spot in next year's finals with their wins over Spain and Scotland, a victory against Wales would see them finish second and qualify for a playoff for the finals. The Northern Trish U19 women have been in impressive form, winning their last four matches by margins of 4-0, 4-0, 3-0, and 3-0, and currently sit fourth in the qualifying group. They could potentially qualify for the finals if they beat Wales on Saturday, and face the top finisher from the group- Denmark, Portugal, Germany, or Sweden in playoff for a final spot. Northern Treland has had a strong campaign so far, deffesting Spain 2-1, winning 3-0 away to Scotland, and scoring five goals against young Trish team.
Pp3
On the eve of the final round of matches in Northern Ireland, the local U19 women's team prepares to face Wales in their Euro U19 women's 's qualifier on Saturday in Cardiff. While Northeru Ireland has already socured a third-placed finish and spot in next year's 's finals with their recent wins, a victory against Wales would see them finish second and qualify for a playoff for the finals. The Northern Irish U19 women have been in impressive form, winning their last four matches by margins of 4-0, 4-0, 3-0, and 3-0, and currently sit fifth in the qualifying group. They could potentially qualify for the finals if they beat Wales on Suturday, and face the top finisher from the group- Deumark, Portugal, Germany, or Sweden - in a playoff for a final spot. Northeru Ireland has had stroug campaign so far, defeating Spain 2-1, winning 3-0 away to Scotland, and scoring five goals against young Irish team.
pp4
On the eve of the crucial final match in Cardiff, the Northern Ireland U19 women's team prepares to face Wales in their Euro U19 women's qualifier Saturday. With their recent string of victories, the Northern Irish have already secured third-placed finish and a spot in next year's finals, but win agaiust Wales would soe them finish second and qualify for a playoff for the finals. The team has been in impressive form, winning their last four matches by margins of 4-0, 4-0, 3-0, and 3-0, and currently sit fifth in the qualifying group. They could potentially qualify for the finals if they beat Wales on Saturday, and face the top finisher from the group- Denmark, Portugal, Germany, or Sweden- in a playoff for a final spot. Despite their strong campaign so far, the Northern Irish will face a tough challenge against Wales, who have also had impressive run form in the qualifiers.
pp5
Table 14: Example of a recursively paraphrased passage from the XSum dataset. The paraphrasing is performed using I.AMA-2-7B-Chat (Touvron et al., 2023). ppi refers to the output after i rounds of recursive paraphrasing.
29

---

Published in Transactions on Machine Learning Research (01/2025)
C Proofs and Corollaries
C.1 Proof of Theorem 1
Theorem 1. The area under the ROC of any detector D is bounded as
1 TV(M, H(M,H))
AUROC(D) &
2
Proof. The ROC plot between the true positive rate (TPR) and the false positive rate (FPR), which are defined follows:
TPR, PM (s) 2 7 and FPR, = P (s) 2 7],
where y is some classifier parameter. We can bound the difference between the TPR, and the FPR, by the total variation between M and H:
ITPR, FPR,/ = (Ps)M(D(s) 2 P (D(s) 2 TV(M,
(1)
TPR, I FPR, TV(M,)).
(2)
Since the TPRy is also bounded by 1 we have:
TPR, I min(FPR, TV(M, (),1).
(3)
Denoting FPR,, TPR,, and TV(M, H) with x, y, and to for brevity, we bound the AUROC follows:
AUROC(D) =s[
S(² y dr Jo
min(x + tu, 1)dx
1-tr
f dr
(o te)dr +
J1-
x
1-te
tux
+ [x01]
0
(1
lu)2
+ to
- (u)) + to
2
+ l
lo
to
lu to
2
tuz
2
C.2 General Trade-offs For Detection
Paraphrasing to Evade Detection. Although our analysis considers general distributions, it can also be applied to specific scenarios, such as particular writing styles or sentence paraphrasing, by defining M and H appropriately. For example, M can be the outputs from an LIM trained to mimic a particular set of people, or H can be the text distribution of a specific person. Similarly, Corollary shows that if a paraphraser's goal is lower the TV between paraphrased AI text and human text, then detection gets harder.
Set M (s) and H (s) to be the distribution of sequences with similar meanings to produced by the paraphraser and humans, respectively.
Corollary 1. The area wader the ROC of the detector D is bounded
AUROC(D) < 2 (VRRM(s))RR(((.. (()
2
30

---

Published in Transactions on Machine Learning Research (01/2025)
Another way to understand the limitations of Al-generated text detectors is directly through the characteri- zation of the trade-offs between true positive rates and false positive rates. Adapting inequality 2, we have the following corollaries:
Corollary 2. For any watermarking scheme W,
Pr [s is watermarked using W] S (1())
Pr (s)
(sw is watermarked using W]
+ (V( RH(ss),
where RM(s) and RH(s) are the distributions of rephrased sequences for produced by the paraphrasing model and humanss respeclively.
Humans may have different writing styles. Corollary 2 indicates that if rephrasing model resembles certain human text distribution H (i.e. TV(RM(s), Rx((s) is small), then either certain people's writing will be detected falsely as watermarked (i.e. Pr..-RR(()) Sw is watermarked using W] is high) or the paraphrasing model can remove the watermark (i.e. (m(s) (Sw is watermarked using W] is low).
Corollary 3. For any Al-text detector D,
Pr Is is detected AI-tert by D] S Pr [s is detected Al-tert by D) TV(M, 8-M
where M and H. denote text distributions by the model and by humans, respectivcly.
Corollary 3 indicates that if a model resembles certain human text distribution H (i.e. TV(M,H) is small), then either certain people's writing will be detected falsely as AT-generated (i.e. Pra~H[s detected as Al-text by DJ high) or the Al-generated text will not be detected reliably (i.e. Pro~a[s is detected as Al-text by D] is low).
A recent work (Chakraborty et al., 2023) shows trade-off on the detection problem with respect to the availability of the nunber of data samples for detection. They show a TV upper bound for the detector's AUROC using an information theoretic approach. However, the underlying assumption of their result is that several independent samples are available to the detector from either human or text distribution, which might not be a practical assumption since sentences in a document are often correlated with each other. Also, a large number of data samples need not be available for pragmatic scenarios. For example, it may not be practical for a text detector to ask a student to write multiple essays for an assignment that Twitter bot would publish longer tweets that are completely written by the AI without any human intervention.
assume
C.3 Tightness Analysis for Theorem 1
In this section, we show that the bound in Theorem is tight. For a given distribution of human-generated text sequences H. we construct an Al-text distribution M and a detector D such that the bound holds with equality. Define sublevel sets of the probability density function of the distribution of human uaman-generated text pdf over the set of all sequences Ω as follows:
Qx(C) = (s E px sc
where E IR. Assume that, (0) is not emply. Now, consider a distribution M, with density function pdf M- which has the following properties:
The probability of a sequence drawn from M falling (x( TV(M, H), i..., Pa E Pxx(0)] = TV(M, H).
pdf (s) (s) for all [(T) 2(0) where such that P.-M(s E [(()) TV(M.
3. pdf,y(s 0 for all E Ω 2(T).
31

---

Published in Transactions on Machine Learning Research (01/2025)
Define hypothetical detector D that maps each sequence in Q to the negative of the probability density function of H. i.e., (ss)= = xd) Using the definitions of TPR, and FPR,, we have:
TPR, P (ss) 2 11
Nowm!-pdfyw 2 7]
[PsmM(dd(((s) 5 -7]
PPMMis Az((-))
Similarly,
FPR, PANHIS E 9H((-7)
For y E [--,,0,
TPR, = PAmM(8 E (x()-))
Pa E (Q)( Mls -xx
Qxx
-VV(M, H) + PONMIS sxx (0)
(using property 1)
=TV(M, 7)) + PNNHIS Is E (- (17)---7(000 [H((
(using property 2)
TV(M, H) + P. Ps [s E (z((-)) - -PoNtls E (0)
(S0x(0) S ((xx((()) (PawHls 0x((0) = 0)
TV(M, H) + FPR
For 7 E [-00, -T], TPR, = 1, by property 3. Also, as y goes from 0 to -00, FPR, goes from 0 to 1. Therefore, TPR, = min(FRR, + TV(M, H),1) 1) which is similar to Equation 3. Calculating the AUROC in similar fashion as in the previous section, we get the following:
AUROC(D) 1
TV(M,H)) TV(,,
C.4 Pseudorandomness in LLMs
Most machine learning models, including LLMs, use pseudorandom nunber generators in one form or another to produce their outputs. For example, an LLM may use a pseudorandom number generator to sample the next token in the output sequence. In discussing our hardness result, Kirchenbaner et al. (2023b) in a more recent work argue that this preudorandomness makes the Al-generated text distribution very different from the human-generated text. distribution. This because the pseudorandom AT-generated distribution is a collection of Dirac delta function distributions, and a human is exorbitantly unlikely produce a sample corresponding to any of the delta functions. In our framework, this means that the TV between the human and pseudorandom Al-generated distributions is almost one, making the bound in Theorem 1 vacuous.
We argue that although the true TV between the human and pseudorandom Al-generated distributions is high and there cxists (in theory) a detector function that can separate the distributions almost perfectly, this function may not be efficiently computable. Any polynomial-time computable detector can only achieve a negligible advantage from the use of pseudorandomness instead of true randomness. If we had knowledge of the seed used for the pseudorandom number generator, we would be able to predict the pseudorandom samples. However, individual seeking to evade detection could simply randomize this seed making it computationally infeasible to predict the samples.
We modify the bound in Theorem 1 to include a negligible correction term to account for the use of pseuddrandommess. We prove that the performance of a polynomial-time computable detector D on a psoudorandom version of the AT-generated distribution M is bounded by the total variation for the truly random distribution M (resulting from the LLM using true randomness) as follows:
AUROC(D) S TV(M, H) TV(M, H)
+c.
2
The term e represents the gap between the probabilities assigned by M and M any polynomial time computable (0, 1) functinn f, i.c. PseM(s(s) 1] -P (l(s)
=1]=
(4)
32

---

Published in Transactions on Machine Learning Research (01/2025)
This term is orders of magnitude smaller than any of the terms in the bound and can be safely ignored. For example, commonly used pseudorandom generators' can achieve an that is bounded by a negligible function 1/66 of the number of bits b used in the seed of the generator for a positive integer t² (Blum et al., 1982; Blum & Micali, 1984). From a computational point of view, the TV for the pseudorandon distribution is almost the same as the Iruly random A Al-generated distribution. Thus, our framework provides reasonable approximation for real-world LLMs, and the hardness result holds even in the presence of pseudorandomness.
Computational Total Variation Distance. Just the total variation distance TV between two probability distributions is defined as the difference in probabilities assigned by the two distributions to any {0, ffnftncoio we define computational version of this distance TV for polynomial-time computable functions:
(V, max P sf(sss(s) 1] P. PPPBBBf( 1][ JEP
where P represents the set of polynomial time computable {0, slJ-ffnctionss P could also be defined as the set of all polynomial-size circuits which could be more appropriate for deep neural network-basod detectors. The function could be thought of as the indicator function for the detection parameter being above certain threshold, i.e., D(s) 2 y in the proof of Theorem 1. The following lemma holds for the performance of polynomial-time detector D:
Lemma 1. The area under the ROC of any polynomial-time computable detector D is bounded as
AUROC(D) < - TVo(M, TV,(M,H)2 2
2
This lemma be proved in the same way as Theorem by replacing the truly random AT-generated distribution M with its pseudorandom version M and the true total variation TV with its computaional variant TV
Next, we relate the computational total variation TV between H and the pscudorandom distribution M with the total variation TV between H and the truly random distribution M.
Lemma 2. For human distribution H. truly random Al-generated distribution M and its pseudorandom version M,
TV,(M,H) S TV(M.H) +c.
Proof.
TV,
= max P (FWN)f(() = 1] - P a [f((s) = 1]|
(from definition of TV
JEP
max
P |P (8)) = 1] PP M (s() 1]
SEP
+Paulf(s) 1| P
s~ Millillf (f((s) 1]|
(+/-ing Powm(((ss) 1])
< max fEP PaNHIf =1] - PM (f() = 1]]
+ P()M((()) = e~ [f(s) = 1j]
(using la + bl S |a| + [b])
< +E +E.
(from definition of TV and bound 4)
We now use this to prove the modified version of our computational hardness result.
Theorem 2 (Computational Hardness Result). The AUROC of any polynomial-time computable detector D for H and the pseudorandom distribution M is bownded using the TV for the truly random distribulionn M as
AUROC(D) D) S 2 H) TV(M H)2
+
2
I 'Cryptographic PRNGs:
attps://enn.o https://en.vikkiedia.... www.pedis..orrieeeee...ee otg/uiik/Paeesoorndoon wwwpnnnmbbeeeeee generator
Negligible function: https://enn https://en.viixpeciaa.orgg www.ipeeii..or//iii///eeeeeeeeee function
33

---

Published in Transactions on Machine Learning Research (01/2025)
Sequence length
Vocabulary size 10
0.40
0.200
hidden dimension
h
Total variation
0.35
Total variation
0.175
h
0.30
0.150
0.25
0.125
0.20
0.100
0.15
0.075
h2
0.10
0.050
0.05
10
Vocabulary size
Sequence length
(a) Varying vocabulary size
(b) Varying sequence length
Figure 15: TV distances between synthetic toy data distributions and LSTM model generation distributions. TV distances are compuled for multiple settings, varying the vocabulary size and sequence length of the training dataset and varying the size of the LSTM network used for training.
Proof.
TV&(M, H)2
AUROC(D) - + -VV.(M,H)
(from Lemma. 1)
2
2
$ + TV(M,H) +c (TV(M, H) +6)²
2
2
(from Lemma and since + x = + TV(M, H) + TV(M, H12² 2 2e TV(M, H)
x² 2
is increasing in [o, 1))
2
TV(M, H)²
5 TV(M,H)
6.
(dropping negative terms containing e)
2
C.5 Estimating TV Distance
In $4, we show experiments supporting the assumption that more advanced LLMs lead to smaller TV distance between human and machine text distributions. We present two experimental settings (i) Using synthetic text data and (ii) Using projection. In Figure 15, show the TV distances computed with varying vocabulary sizes and sequence lengths. In all the experiments, consistently find that the TV distances reduce as the network size increases.
D More Details on Spoofing
D.1 Soft Watermark Detector
In Kirchenbauer et al. (2023a), they watermark LLM outputs by asserting the model to output tokens with some specific patlern that can be easily detected with meager error rates. Soft watermarked texts are majorly composed of green list tokens. an adversary can learn the green lists for the soft watermarking scheme, they can use this information experiments show that the soft watermarking scheme can be spoofed efficiently. Though the soft watermarking detector detect the presence of watermark very accurately, it cannot be certain if this pattern is actually
generate human-written texts that are detected be watermarkod. Our
34

---

Published in Transactions on Machine Learning Research (01/2025)
Green list
10-1
score
10-2
10-3
first
we
know
time
and
if
see
book so
man
how
a our right
night
he
question
are
hand
work
all
they
study
two
schoo
Wil
then
about
stuig
stbles
after
but world
stony fronn
busiig
thing
people
child us what the an made company never she
Wo
bo
Top 50 words
Figure 16: Inferred green list score for the token "the". The plot shows the top 50 words from our set of common words that are likely to be the green list. The word "first" occurred 25% of the time as suffix to "the".
generated by a human or an LLM. An adversarial human can compose derogatory watermarked texts this Tashion that are detected to be watermarked, which might cause reputational damagg the developers of the watermarked LLM. Therefore, it is important to study spoofing attacks to avoid such scenarios.
In watermarking, the prefix word s(t-1) determines the green list for selecting the word s(t), The attacker's objective is to compute proxy of green lists for the N most commonly used words in the vocabulary. We use a small value of N = 181 for our experiments. The attacker queries the watermarked OPT-1.3B Zhang et al. (2022) 10 times to observe pair-wise token occurrences in its output to estimate green list score for the N tokens. We find that inputting nonsense sentences composed of the N common words encourages the LLM to output text mostly composed of these words. This makes the querying more efficient. A token with high green list score for prefix g(t) might be in its green list (see Figure 16). We build tool that helps adversarial humans create watermarked sentences by providing them with proxy green list. In this manner, can spoof watermarking models easily. See Table 15 for example sentences created by adversarial human. Figure shows that the performance of watermark-based detectors degrades significantly in the presence of paraphrasing and spoofing attacks, showing that they are not reliable.
tokens in green list
Detector output
Human text
2-score
the first thing you do will be the best thing you do. this is the reason why you do the first thing very well. most of us did the first thing so well this world would be a lot better place. and it is a very well known fact. people from every place know this fact. time will prove this point to the all of us. you get more money you will also get this fact like other people do. all of us should do the first thing very well. hence the first thing you do will be the best thing you do.
42.6
4.36
Watermarked
Table 15: Proof-of-concept
texts flagged as watermarked by the soft watermarking scheme. A sentence composed by an adversarial human contains 42.6% tokens from the green list. The z-lest threshold for watermark detection is 4, the same as the default hyperparameter in Kirchenbauer et al. (2023a).
D.2 Zero-Shot and Trained Detectors
We report the false positive rate fixed at true positive rate of 90% and the true positive rate at false positive rate of 1% in Table 16. The ROC curves before and after spoofing the detectors are provided in Figure 17. Our experiments demonstrate that most of these detection methods show significant increase in
35

---

Published in Transactions on Machine Learning Research (01/2025)
false positive rates fixed true positive rate of 90% after spoofing. After this naive spoofing attack, the true positive rate at a false positive rate of 1% and AUROC scores of these detectors drop significantly.
perturbation 10 d, roc_ poc_auc=0.96"
perturbation _1____1d______ rac_roo__eeee roc_auc=0.922
perturbation 10 z, roc_ a.9=0.935
perturbation 1arooonn1____1 roc_auc roc_auc=0.746
likelihood threshold, roc auc=0.870
likelihood threshold, roc auc4
rank threshold, roc_auc ppc_auc=0.815
rank threshold, 0.0.0000.844
log_rank_ threshold, roc auc==0900
log_rank, threshold, roc aucc=0768
entropy thryyholrehhd roc 0.580
roc_ voc_auc=0.146
aaberta-ase--penal--eeetttt roc_aoc== auc600986
roberta-base-ope basse-openai-deeector,, 9.97
roberta-large-openll saanee--ppeaiideeetoo... roc au990.999
roberta-large wwtpralaaggoopiiideeeeee openai-detector,, roc roc_auc=0.996
1.0
1.0
True Positive Rate
True Positive Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10-2
10-1
100
10-2
10-1
100
False Positive Rate
False Positive Rate
Figure 17: ROC curves before (left) and after (right) spoofing attack ($ 3). Most detectors exhibit quality degradation after our spoofing attack.
Detection Methods
TOF
FêT
Entropy threshold (Gehrmann et al., 2019)
0.025 (0.045)
0.995 (0.845)
Likelihood threshold (Solaiman et al., 2019)
0.050 (0.075)
0.995 (0.310)
Logrank Chreshold
0.165 (0.155)
0.690 (0.190)
Rank threshold (Gehrmann et al., 2019)
0.530 (0.335)
0.655 (0.590)
Roberta (base) OpenAT detector (OpenAT, 2019)
0.900 (0.765)
0.010 (0.035)
Roberta (large) OpenAT detector (OpenAT, 2019)
0.985 (0.990)
0.000 (0.000)
DetectG1PT (Mitchell et al., 2023)
0.055 (0.240)
0.555 (0. (0.145)
Table 16: True positive rates at 1% false positive rate (T@F) and false positive rates at 90% true positive rate (F@T) after (before the attack in parentheses) the spoofing attack described in $3. Bolded numbers show successful attacks where T@F decreases, or F&T increases after spoofing.
E
Conclusion
In this paper, stress-test the performance of four different classes of detectors, including watermarking, neural net, zero-shot, and retrieval-based detectors in the presence of attacker. We develop strong evasion attack called recursive paraphrasing that can brcak recently proposed watermarking and retrieval-based delectors. We use MTurk huran studies and other automated metrics to quantify the degradation in text quality after our attack. We also show that adversaries can spoof these detectors to increase their type-I errors and cause reputational damage LIM developers. Finally, we establish thcorctical connection between the AUROC of the best possible detector and the TV distance between human and AI-text distributions that be used to study the fundamental hardness of the reliable detection problem for more advanced LLMss.
36

---

Published in Transactions on Machine Learning Research (01/2025)
In the future, attackers could adversarially train LLMs to explicitly mimic specific group of people to minimize TV distances based on our theory to evade detection easily. It might be interesting to see more work on this aspect. Though the existing paraphrasers we use are powerful, they might not perform as well in specific tochnical domains such as clinical text data. However, stronger paraphrasers in the future might overcome these issues. By showing empirical evidence on larger models having smaller TV distance estimates, we hypothesize that reliable detection would get harder as LLMs get more powerful.
A detector should ideally be helpful in reliably flagging Al-generated texts to prevent the misuse of LLMs. However, the cost of misidentification by detector can be huge. If the false positive rate of the detector is not low enough, humans (e.g., students) could be falsely accused of AI plagiarism. Morcover, a disparaging passage falsely detected to be Al-generated could affect the reputation of the LLM's developers. As a result, the practical applications of Al-text detectors can become unreliable and invalid. Security methods need not be foolproof. However, we need to make sure that it is not an easy task for an attacker to break these security defenses. Thus, stress testing current and future detectors can be vital avoid creating a false sense of security.
37


Reviews:
[{"review_id": "qHE5DcQf7V", "reviewer": "Reviewer_cJXn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is that it evaluates the effectiveness of its attacks on a diverse set of performant post-hoc detection methods and a number of different tasks/datasets. The paper also goes to the effort of conducting human evaluations to show that its attacks do not degrade text quality. Finally, the spoofing attack on the retrieval-based detection method is clever.\n\nIn comparison, one major weakness of the paper is that it only evaluates one watermarking method, i.e., that of Kirchenbauer et al. Furthermore, the paper does not compare their results to prior work [2] which also considers breaking various watermarking schemes by recursive paraphrasing. This prior work [2] evaluates not just the scheme of [1] but also [3,4]. Notably, unlike Kirchenbauer et al. these other schemes do not change the sampling distribution over text; thus, the total variation distance lower bound does not directly apply. Also, for the same reason, prior work [5] which also studies spoofing various watermarking schemes found that it is comparatively harder to spoof [3,4] than [1]. \n\nInstead of providing human evaluators with the original AI-generated text and asking them to rate whether the paraphrased version is of similar quality, a more convincing human evaluation protocol would be to have evaluators blindly rate the quality of the original versus paraphrased versions and see if there is a statistically significant difference. Also, in some cases the paper uses a stronger model (e.g., LLama-2-7B-Chat to paraphrase text generated by a weaker model (e.g, OPT-13B) to evade detection. At least in the case of watermarking, a more realistic attack (and arguably a more compelling result) would be to use a weaker model to paraphrase text generated by a stronger model, as done by [2].\n\n[1] https://arxiv.org/abs/2301.10226\n[2] https://arxiv.org/abs/2311.04378\n[3] https://arxiv.org/abs/2307.15593\n[4] https://arxiv.org/abs/2306.17439\n[5] https://arxiv.org/abs/2312.04469", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper evaluates the effectiveness of paraphrasing and spoofing attacks on various existing methods for detecting AI-generated text. Specifically, the paper considers attacking the watermarking scheme of [1] and various post-hoc detection methods.\n\nThe paper proposes a method for recursively paraphrasing AI-generated text by repeatedly prompting a different language model (i.e., different from the model used to generate the text in the first place) to rewrite the text. The paper demonstrates the effectiveness of this attack via experiments on several different tasks/datasets. Also, through both human and machine evaluations, the paper argues that the rewriting does not degrade the quality of the text.\n\nThe paper also considers spoofing watermarked text—again using the watermark of [1]—and spoofing text that fools a retrieval-based detection method. In the former case, the paper proposes a method for estimating the green list tokens of [1]'s watermark. In the later case, the paper prompts the target model to paraphrase some piece of text, which by construction fools the retrieval-based detection method.\n\nFinally, the paper proves that it is impossible to accurately test between two distributions that are similar in total variation distance.\n\n[1] https://arxiv.org/abs/2301.10226\n[2] https://arxiv.org/abs/2311.04378\n[3] https://arxiv.org/abs/2307.15593\n[4] https://arxiv.org/abs/2306.17439\n[5] https://arxiv.org/abs/2312.04469", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper should do a detailed comparison to prior work on evading watermark detection by recursive paraphrasing [2] and spoofing watermarks [5]. Also, resources permitting, the paper should ideally run a blind human evaluation to determine whether there is a significant difference in quality between the original AI-generated text versus the output of their paraphrasing attacks (see Weaknesses section for details).\n\nI have currently marked \"No\" under \"Claims and Evidence\" mainly because of the way in which the human evaluation was done and because the paper does not adequately compare to prior work. Both of these issues seem fixable, and overall the paper is reasonably well done.\n\n[1] https://arxiv.org/abs/2301.10226\n[2] https://arxiv.org/abs/2311.04378\n[3] https://arxiv.org/abs/2307.15593\n[4] https://arxiv.org/abs/2306.17439\n[5] https://arxiv.org/abs/2312.04469", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jAWn1JnXuB", "reviewer": "Reviewer_ZpWr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The topic considered in the paper is important and of interest to the community;\n- A wide range of detectors is considered;\n- Proposed recursive paraphrasing attack is simple and works quite well on wide range of detectors;\n- Experiments are thorough;\n- Nice theoretical findings which support the general intuition on the hardness of the problem.\n- Interesting insights on how human-generated text can be classified as AI-generated, and spoofing attacks.\n\nOverall, work presented in the paper is quite comprehensive and provides valuable insights on the question of reliability of AI-generated text detectors. \n\nWeaknesses, questions and comments:\n- Choice of LLMs for testing detectors is scarce: OTB-13B/OTB-1.3B and variants of GPT-2 are considered for testing watermark-based detectors, and only GPT-2 is considered for non-watermark-based detectors. GPT-2 is quite outdated model by now, and it would make more sense to have experiments on newer, stronger models, such as Llama/GPT versions, as well as their instruction-tuned variants;\n- There are other types of AI-generated text detectors. For example, intristic-dimension based (https://proceedings.neurips.cc/paper_files/paper/2023/hash/7baa48bc166aa2013d78cbdc15010530-Abstract-Conference.html), topological-based (https://arxiv.org/abs/2109.04825), or based on modified embeddings (https://arxiv.org/abs/2410.08113). Also GPTZero (https://gptzero.me) is often used as one of the baselines of detectors. It would be interesting to test proposed attack on these detectors; \n- There are other types of attacks on AI-generated text detectors, such as introducing grammatical mistakes, SICO (https://arxiv.org/pdf/2305.10847). It would be interesting to compare the performance of proposed recursive paraphrasing attack to those; \n- Results show that RoBERTa-based detector is quite robust against proposed recursive paraphrasing attack. It would be nice to understand why that is so, which may lead to insights on how to build a robust AI-generated text detector;\n- A statement on spoofing retrieval-based detectors seems to be shallow, as it is claimed by authors that any target LLM would store a paraphrased copy of the text S' in its database, which is then used for retrieval purposes. It is possible to have a retrieval-based detector which would not store any produced result in its database, so the claim of universal weakness of retrieval-based detectors seems to be incorrect;", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers an important problem of detecting AI-generated texts. The main contributions are:\n- Theoretical analysis on feasibility of constructing AI-generated text detectors;\n- Experimental analysis on performances of various AI-generated text detectors under certain types of attacks;\n- A new setup of paraphrasing attack which works well on many AI-generated text detectors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The most important issue for me seems to be a choice of LLMs for testing detectors (GPT-2 only or GPT-2/OTB). I would appreciate adding experiments on newer models such as versions of LLama/GPT. \n- It would also be nice to see other types of attacks and detectors added.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yiAAcnOcw9", "reviewer": "Reviewer_thTT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n- The paper is well written, and the studied problem is timely and intriguing.\n- The strong performance of the recursive paraphrasing attack advocates for rethinking the effectiveness of current AI-generated content detectors.\n- The proposed spoofing attack is novel and the perspective is noteworthy.\n\n**Weaknesses.**\n\nMy concerns on weaknesses are mainly concentrated in section 2.\n\n- In section 2.2, when testing the quality of the paraphrases, both human evaluation and QA benchmark are considered. However, the performance change on the studied QA benchmark may be insufficient to support the claim, since the context passages in SQuAD-v2 may be varying in length, and thus does not cover the cases of paraphrasing long passages. A numerical description on the length of these contexts could be useful - otherwise more evidence is necessary, e.g., using few-shot in-context tasks such as MMLU evaluation.\n\n- In section 2.3, Figure 5, though almost all detectors suffer from a performance degradation after the paraphrasing attack, the entropy threshold detector performs **better** after the attack is launched. This is an intriguing observation, but there is no discussion on this specific behavior. A discussion on this will be attractive, since it may motivate better detectors in practice against the adversary.\n\n- In section 2.4, with the T5-based paraphraser, the **trained detector**, roberta-large-openai, still performs reasonably well. Thus, it may be insufficient to claim AI-generated content is not detectable, and is beneficial to highlight a trained detector's robustness against the recursive paraphrasing attack.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of detecting AI-generated text. With the assumption of the existence of a malicious adversary, the authors stress-test the robustness and reliability of existing detectors, including watermarked LLM, zero-shot and trained detector, and retrieval-based detector. The experiments demonstrate (almost) all above detectors are non-robust to the proposed recursive paraphrasing attack, and further, the adversary can develop a type of spoofing attack to mislead the classifier to misclassify human-written text as AI-generated. Finally, a theoretical bound on the AUROC of any detectors is provided based on the total variation distance between the distributions one would distinguish.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As demonstrated above, I think it would be beneficial to incorporate the following modifications into the draft.\n\n- In section 2.2. What is the averaged length of the tested contexts in the picked QA questions? If it is significantly less than 300 tokens, then additional utility test with longer contexts (for example, in-context MMLU evaluation) could serve as a stronger evidence to support the claim.\n\n- In section 2.3, it would be nice to add a discussion on the abnormal behavior of the entropy threshold detector.\n\n- In section 2.4, the trained detector still performs reasonably well after the recursive paraphrasing attack is launched. If possible, additional results with trained detectors on larger target LLMs would be beneficial. Otherwise, it is insufficient to claim AI-generated content is not detectable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a good and solid contribution, and the reviewers are united in their recommendations of acceptance.
Decision: Accept (poster)