SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kalpesh Krishna\* Yixiao Song\* Marzena Karpinska\* John Wieting\* Mohit Iyyer\*
$\spadesuit$  University of Massachusetts Amherst,  $\diamond$  Google,  $\diamond$  Google DeepMind {kalpeshk, jwieting} @ google.com yixiaosong@umass.edu {mkarpinska,miyyer}@cs.umass.edu
The rise in malicious usage of large language models, such as fake content creation and academic plagiarism, has motivated the development of approaches that identify AI-generated text, including those based on watermarking or outlier detection. However, the robustness of these detection algorithms to paraphrases of AI-generated text remains unclear. To stress test these detectors, we build a 11B parameter paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking, GPTZero, DetectGPT, and OpenAI's text classifier. For example, DIPPER drops detection accuracy of DetectGPT from  $70.3\%$  to  $4.6\%$  (at a constant false positive rate of  $1\%$ ), without appreciably modifying the input semantics.
To increase the robustness of AI-generated text detection to paraphrase attacks, we introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider. Given a candidate text, our algorithm searches a database of sequences previously generated by the API, looking for sequences that match the candidate text within a certain threshold. We empirically verify our defense using a database of 15M generations from a fine-tuned T5-XXL model and find that it can detect  $80\%$  to  $97\%$  of paraphrased generations across different settings while only classifying  $1\%$  of human-written sequences as AI-generated. We open-source our models, code and data.<sup>1</sup>
Large language models (LLMs) such as ChatGPT [Schulman et al., 2022] exhibit an unprecedented ability to write coherent and relevant long-form text in response to user-specified prompts. These abilities have sparked fears of malicious applications such as automatically generating fake news articles or homework answers [Stokel-Walker, 2022]. To defend against these use cases, several algorithms have recently been proposed to detect AI-generated text, including watermarking [Kirchenbauer et al., 2023a], GPTZero [Tian, 2023], DetectGPT [Mitchell et al., 2023], and OpenAI's text classifier [OpenAI, 2023a]. However, it remains unclear how robust these algorithms are to paraphrase attacks, in which AI-generated text from an LLM is rewritten by another (smaller) model to convey approximately<sup>2</sup> the same meaning but using different word choice and syntax.
*Work done as a PhD student at UMass, and partially as a student researcher in Google Research.
$\dagger$  John Wieting and Mohit Iyyer contributed equally as advisors.
<https://github.com/martiansideofthemoon/ai-detection-paraphrases>
2We use the quasi-paraphrase definition of semantic equivalence [Bhagat and Hovy, 2013] in this paper.
Figure 1: An overview of paraphrasing attacks with DIPPER on watermarked text [Kirchenbauer et al., 2023a]. The original model generation (top) contains several "green" watermarked tokens that are counted by a detector to judge whether the text was AI-generated. After paraphrasing, several green tokens are replaced with approximately semantically-equivalent red tokens, thereby fooling the detector. Actual outputs from a watermarked version of GPT2-XL and our paraphraseer DIPPER.
In this paper, we first demonstrate the vulnerability of these existing detectors to paraphrase attacks (Section 3, 4). Such attacks require an external paraphrase model, since paraphrases generated by the base LLM are still susceptible to detection techniques such as watermarking. We train an 11B parameter paraphrase generation model called DIPPER (or Discourse Paraphraser) to execute these attacks. DIPPER possesses two unique features that help its outputs evade AI-generated text detectors: (1) Paraphrasing long-form text in context: Most modern paraphrasers are exclusively trained on sentence-level data, ignoring discourse-level information. However, many critical use cases of LLMs involve generating long-form text as responses to detailed user-specified prompts. Thus, we train DIPPER to paraphrase paragraph-length texts, re-order content, and optionally leverage context such as user prompts; (2) Controlling output diversity: Another weakness of existing paraphrasers is that they lack an easy way to control output diversity. An attacker may want to apply just the minimum amount of paraphrasing to evade a detector. DIPPER provides users with two intuitive scalar control knobs at inference time (lexical diversity, content reordering) that are trained end-to-end.
We use DIPPER to attack several recently proposed AI-generated text detectors (see Figure 1 for an attack overview). Experiments on multiple tasks and LLMs (including GPT3.5-davinci-003) show that after paraphrasing with DIPPER, a substantial fraction of AI-generated texts are misclassified as human-written texts by all detectors. For example, DetectGPT [Mitchell et al., 2023] correctly detects  $70.3\%$  of AI-generated sequences from GPT2-XL, but after paraphrasing, its detection rate drops to only  $4.6\%$ $^{3}$  despite minimal semantic modification. We confirm the validity of DIPPER's paraphrases through several automatic evaluations and a human evaluation of semantic similarity.
Given the vulnerability of AI-generated text detectors to paraphrasing, how can we defend against such attacks? In the second part of our paper (Section 5), we propose to use retrieval methods to detect AI-generated text instead of relying on statistical properties of the text or watermarking. First, an LLM API provider stores every output generated by their model in a database. The API provider then offers a service in which a semantic representation of a candidate text is compared to representations of every generation stored in the database. The search focuses on the semantics of the input and can leverage both standard IR methods such as BM-25 [Robertson et al., 1995] as well as semantic vector representations such as P-SP from Wieting et al. [2022]. Since paraphrasing does not modify the semantics of the input, this algorithm is robust to paraphrasing attacks. Specifically, we find that  $97.3\%$  of PG19 paraphrases and  $80.4\%$  of Wikipedia paraphrases are successfully detected in a large database of over 15M generations, at a  $1.0\%$  false positive rate. We extensively discuss the limitations and scalability of retrieval-based detection in Section 5.4.
In contrast to concurrent work that also uses paraphrasing to attack AI-generated text detectors [Sadasivan et al., 2023], our work offers more comprehensive attack experiments, a new and more powerful paraphraseer, human evaluations of paraphrase quality, and finally a novel defense mechanism based on retrieval to combat such attacks. To spur future research in this area, we will release our DIPPER model, data, and a codebase for evaluating both existing detectors and our retrieval-based method.
3These detection rates were computed at a constant false positive rate (FPR) of  $1\%$ . Due to the importance of low FPR in this task, we recommend using a fixed low FPR rather than AUC-ROC values; see Section 4.1.
2 Background on detectors of AI-generated text
In this section, we provide a brief overview of existing algorithms for detecting AI-generated text detection (see Appendix E for a detailed version). We also contrast our work to Sadasivan et al. [2023], a concurrent effort which notes the efficacy of paraphrasing attacks but does not consider a retrieval-based defense in its pessimistic conclusion about the fate of AI-generated text detection.
A watermark is a modification to the generated text that can be detected post-hoc by an algorithm while remaining imperceptible to human readers. Effective watermarks are difficult to remove and have little effect on the quality of generated text. Prior work has watermarked natural language using syntax tree manipulations [Topkara et al., 2005, Meral et al., 2009], and this area has received renewed interest with the advent of LLMs [Abdelnabi and Fritz, 2021, Grinbaum and Adomaitis, 2022]. Most recently, Kirchenbauer et al. [2023a] proposed a simple algorithm that watermarks LLMs by slightly perturbing its probability distribution while generating text.
Statistical outlier detection methods detect AI-generated text based on its artifacts [See et al., 2019, Holtzman et al., 2020] instead of modifying the generative algorithm. Early methods detect statistical irregularities in entropy [Lavernne et al., 2008] and perplexity [Beresneva, 2016], while Gehrmann et al. [2019] introduced the GLTR visualizer to assist humans in detecting AI-generated text. The release of ChatGPT prompted the development of two new tools: closed-source GPTZero [Tian, 2023] and open-source DetectGPT [Mitchell et al., 2023]. The latter uses the observation that AI-generated text tends to have significantly higher LLM likelihood than meaningful perturbations of it.
Classifier methods train models to distinguish human-written text from AI-generated text. Early efforts used classifiers to detect fake reviews [Hovy, 2016] and news [Zellers et al., 2019], while others examined classifier performance across domains [Bakhtin et al., 2019] and decoding strategies [Ippolito et al., 2020]. Most recently, OpenAI fine-tuned a GPT model to perform this task and released it as a web interface [OpenAI, 2023a]. Their model uses generations from 34 LLMs, with the human-written text from Wikipedia, WebText, and their internal human demonstration data.
Comparison to Sadasivan et al. (2023): In recent concurrent work, Sadasivan et al. [2023] also demonstrate the utility of paraphrasing attacks against AI-generated text detectors. Our experiments encompass more tasks, detection algorithms, and larger LMs like GPT3.5. Additionally, we propose a discourse-level paraphrase model (DIPPER) that is much more suited to long-form text than the off-the-shelf sentence-level paraphraseurs used in their paper. More importantly, our retrieval-based defense directly contradicts the "impossibility result" of Sadasivan et al. [2023] and its associated proof, which states that an optimal detector will perform at random as the quality of LLM-generated text approaches that of human-written text. The quality of generated text is irrelevant to our detector's accuracy because it relies only on a corpus search, and thus the proof is inapplicable. Other concurrent work [Chakraborty et al., 2023] has also shown the proof's invalidity in practical settings.
3 Building a controllable discourse paraphraseer
Having outlined existing methods to detect AI-generated text, we now focus on a simple attack against all detection techniques: paraphrasing the generated text. Intuitively, paraphrasing alters the statistical properties of AI-generated text, which can fool outlier detection or classifiers while also reducing the number of watermarked tokens (Figure 1). To evade such detectors, a paraphrase must be able to handle context in the form of prompts or multi-sentence inputs. Its behavior should also be controllable in order to make as many/few changes as needed to evade a given detector. In all cases, it should not appreciably change the input semantics. Finally, to evade watermarking, the paraphrase must be different from the watermarked model, as otherwise the paraphrases will also be watermarked. Below, we detail how we construct a paraphrase (DIPPER) with all these properties.<sup>4</sup>
Constructing paraphrase data: Our process involves fine-tuning a LLM on a parallel dataset of paragraph-level paraphrases, which we modify to model control, external context and content reordering. We leverage the PAR3 dataset [Thai et al., 2022], which contains multiple translations of non-English novels into English aligned at a paragraph level, which we treat as paraphrases. More formally, let  $p$  and  $q$  be aligned paragraphs, where  $p_1, p_2 \ldots p_N$  denote sentences of  $p$  and  $q_1, q_2, \ldots q_M$
To better ground DIPPER's abilities in prior work, we survey existing paraphrase models in Appendix D.1.
Step 1: Align sentences between translation 1 and translation 2 using semantic similarity. alignments  $= \left( {\left( {p1},{q1}\right) ,\left( {{p2},{q2}}\right) ,\left( {{p3},{q3q4}}\right) ,\left( {{p4},{q5}}\right) }\right)$
Step 2: Choose a subset of alignments ((p2, q2), (p3, q3q4))  
Step 3: Shuffle sentences in q, compute control codes
Lexical diversity: 40 / 100
Step 4: Input / output mapping to fine-tune for discourse paraphr.
Figure 2: The method used to train DIPPER on English translations of the French novel *The Nun*. We first align sentences between the two translations to create parallel data. Next, a subset of the alignments are chosen; in this example, we use  $(p_2, q_2)$  and  $(p_3, q_3q_4)$ . We shuffle sentences, compute control codes, and fine-tune a T5-XXL LM to generate  $p_2p_3$  given  $q_3q_4q_2$  and the context  $p_1$  and  $p_4$ .
denote sentences of  $q$ . Note that  $M$  may not be equal  $N$  when two translators disagree on when to merge and split sentences. We perform the following steps (overview in Figure 2):
1. Align sentences of  $p$  to sentences of  $q$  by using the semantic similarity scores from the paraphrase similarity metric in Wieting et al. [2019] to run the sequence alignment algorithm designed by Needleman and Wunsch [1970] which uses dynamic programming (metric details in Section 4.1).
2. Choose a subset of sentences  $p_i \ldots p_j$  from the first paragraph. Let  $q_{i'} \ldots q_{j'}$  be the corresponding alignment in the second paragraph. In Figure 2,  $i = 2, j = 3, i' = 2, j' = 4$ .
3. Re-order the sentences  $q_{i'} \ldots q_{j'}$  randomly, and compute the diversity control codes between  $p_i \ldots p_j$  and shuffle  $(q_{i'} \ldots q_{j'})$ . We shuffle the sentences to allow for the model to learn content re-ordering. We compute lexical diversity  $(L)$  using unigram token overlap (F1 score), and the order diversity  $(O)$  using the Kendall-Tau correlation of tokens of overlapping words between  $p_i \ldots p_j$  and shuffle  $(q_{i'} \ldots q_{j'})$ , also used in Krishna et al. [2020]. These scores are normalized to values  $\{0, 20, 40, 60, 80, 100\}$ , where  $L = 20$  roughly corresponds to a  $20\%$  lexical modification.
4. Map the shuffled  $q_{i'} \ldots q_{j'}$  to  $p_i \ldots p_j$ , leveraging context from the rest of  $p$  and control codes using string concatenation. Let input = shuffle  $(q_{i'} \ldots q_{j'})$ . We map,
$$
\text {l e x i c a l} = L, \text {o r d e r} = O \oplus p _ {1} \dots p _ {i - 1} \oplus <   p > \text {i n p u t} <   / p > \oplus p _ {j + 1} \dots p _ {N} \longrightarrow p _ {i} \dots p _ {j}
$$
where  $\oplus$  is string concatenation. During inference, we can paraphrase any sequence of sentences by marking it with  $<\mathsf{p}>$  tags, assigning the control codes  $(L,O)$  the desired diversity values.
Our final dataset contains 6.3M paraphrase pairs. We fine-tune a sequence-to-sequence Transformer [Vaswani et al., 2017] on this data, initialized with the pretrained 11B parameter T5-XXL checkpoint [Raffel et al., 2020]. See Appendix F.1 for details.
4 Experiments attacking detection algorithms with DIPPER
In this section, we describe our experimental setup in Section 4.1-4.2 and present our results in Section 4.3. Overall, we find that DIPPER evades all detectors across three LLMs (including GPT3.5).
Detection accuracy: Our first metric measures how often the input text is correctly detected as AI-generated. Since detection rates are heavily dependent on the chosen detection threshold, the AUC-ROC metric is commonly used to measure detector performance [Mitchell et al., 2023], which considers the range of all possible thresholds. However, in this application, it is critical that the false positive rate (FPR) is low; in other words, human-written text must almost never be classified as machine-generated [OpenAI, 2023a, Kirchenbauer et al., 2023a]. Hence, we fix the FPR to  $1\%$  for all detection algorithms (although even  $1\%$  is likely too high in practice), and adjust the detection threshold accordingly while reporting detection accuracies. Additionally, we also plot ROC curves focusing on the  $0 - 1\%$  FPR region. Overall, we expect detection rates to plummet on paraphrased text.
Table 1: Performance of detection algorithms (at  $1\%$  FPR) before and after DIPPER paraphrasing on open-ended generation using Wikipedia prompts (300 generated tokens). As the diversity (L,O) increases, detection rates decrease across algorithms, with nearly perfect semantic similarity (Sim). *GPT3.5 DetectGPT scores computed using 200 samples at  $20\%$  FPR as it scores  $0\%$  at a  $1\%$  FPR.
Metric → Detector →	Sim ↑	Detection Accuracy ↓
Watermarks	DetectGPT	OpenAI	GPTZero	RankGen
GPT2-1.5B	-	100.0	70.3	21.6	13.9	13.5
+ DIPPER 20L	99.2	97.1	28.7	19.2	9.1	15.8
+ DIPPER 40L	98.4	85.8	15.4	17.8	7.3	18.0
+ DIPPER 60L	96.9	68.9	8.7	13.3	7.1	19.8
+ DIPPER 60L, 60O	94.3	57.2	4.6	14.8	1.2	28.5
OPT-13B	-	99.9	14.3	11.3	8.7	3.2
+ DIPPER 20L	99.1	96.2	3.3	11.8	5.4	5.2
+ DIPPER 40L	98.6	84.8	1.2	11.6	3.8	6.6
+ DIPPER 60L	97.1	63.7	0.8	9.1	6.3	9.3
+ DIPPER 60L, 60O	94.6	52.8	0.3	10.0	1.0	13.5
GPT-3.5-175B, davinci-003	-	-	26.5*	30.0	7.1	1.2
+ DIPPER 20L	97.6	-	12.5*	20.6	4.3	1.7
+ DIPPER 40L	96.7	-	8.0*	22.4	4.8	2.0
+ DIPPER 60L	94.2	-	7.0*	15.6	6.1	3.9
+ DIPPER 60L, 60O	88.4	-	4.5*	15.6	1.8	7.3
Human Text	-	1.0	1.0	1.0	1.0	1.0
Semantic similarity (Sim): Detection accuracy is an insufficient evaluation of our attack's success. We also need to measure whether the original and paraphrased generations share approximately the same semantics. We measure semantic similarity using the state-of-the-art semantic similarity model P-SP from Wieting et al. [2022], an embedding averaging model trained on a large corpus of filtered paraphrase data [Wieting and Gimpel, 2018]. P-SP is a well-calibrated metric that performs well on semantic calibration tests as well as plagiarism detection in STS benchmarks [Agirre et al., 2016]. P-SP is also robust against topically similar non-paraphrases. We found that P-SP it scores just 0.09 on random pairs of paragraphs from the same book (topically similar paragraphs but not paraphrases) in the PAR3 dataset [Thai et al., 2022]. In contrast, the average P-SP score of actual human paraphrase pairs in PAR3 is 0.76. We consider semantics to be approximately preserved if the P-SP score is greater than this average human paraphrase score of 0.76.
Besides semantic similarity, we conduct several automatic evaluations, ablation studies, and human evaluations of intrinsic paraphrase quality in Appendix C.
4.2 Models, datasets & detection algorithms
Base language models: We conduct attacks on three language models of varying sizes that belong to different model families. We consider the GPT2-XL model (1.5B parameters) [Radford et al., 2019], the OPT-13B model [Zhang et al., 2022], and the text-davinci-003 variant from the GPT-3.5 family [Brown et al., 2020], which has 175B parameters and has additionally been instruction tuned using reinforcement learning from human feedback [Ouyang et al., 2022]. For all LMs, we sample generations that are 300 tokens long before passing them through DIPPER for the attack experiments.
Natural language generation tasks: We experiment with two long-form text generation tasks, since most malicious applications (e.g., fake article creation) are associated with long-form outputs. First, we consider open-ended generation, where an LM generates a continuation to a two-sentence prompt. To obtain our prompts, we sample 3K contiguous two-sentence chunks from the validation split of WikiText-103 [Merit et al., 2017] and use the next 300 tokens as the "human-written" continuation. Second, we evaluate long-form question answering [Fan et al., 2019], in which an LM answers a question with a 300-word answer (dataset details in Appendix F.2). For our main results, the human
For GPT2-XL and OPT-13B, we generate text using nucleus sampling [Holtzman et al., 2020] with  $p = 0.9$ . For davinci-003, we use the default hyperparameters on the API Playground (temperature = 0.7).
reference answers or continuations are only used to adjust detection thresholds of studied methods to maintain a  $1\%$  FPR. Note that we are not removing human-written text from our test set. Our metric is equivalent to having a test set with a 50-50 split between machine/human-written text for the same prompts, and observing the  $\mathrm{FPR} = 1\%$  point in the ROC curve (also provided in Appendix H).
Detection algorithms: We attack five detectors: (1) watermarking [Kirchenbauer et al., 2023a]; (2) DetectGPT [Mitchell et al., 2023]; (3) GPTZero [Tian, 2023]; (4) OpenAI's text classifier [OpenAI, 2023a]; and (5) RankGenXL-all [Krishna et al., 2022a]. We use the default hyperparameters for each detector. We also respect their minimum length specifications, discarding instances where any of the AI-generated text, human-written text, or paraphrased text is shorter than the minimum length.
Paraphrasing AI-generated text: We pass the prompts for each task and AI-generated responses to those prompts through DIPPER. Specifically, we feed the model input of the form,
lexical  $=$  L,order  $= 0$  prompt  $\langle \mathbb{P}\rangle$  generated-text  $\langle \mathbb{P}\rangle$
where  $L$  and  $O$  represent the paraphrase. diversity control codes and the  $\langle \mathfrak{p}\rangle$  and  $\langle \mathfrak{p}\rangle$  special tokens mark the boundaries of the text to be paraphrased. We use  $L = 20,40,60$  and  $O = 0,60$  in our main attack experiments. After paraphrasing, we ensure that the AI-generated sequence, paraphrased sequence, and human-written sequence have an equal number of words by truncating them to the length of the shortest among the three. To ensure higher semantic preservation, we iteratively paraphrase long sequences three sentences at a time, keeping al
ready paraphrased text in the context of the generation. To show the effectiveness of our attack, we only paraphrase each generation once, rather than draw multiple samples until it evades detection. $^{10}$
Figure 3: Detector performance (at  $1\%$  FPR) on long-form QA before/after paraphrasing. As diversity (L,O) increases, detection rates decrease with very high semantic preservation (Sim). WM: Watermark, D.GPT: DetectGPT, O.AI: OpenAI. *GPT3.5 D.GPT uses 100 samples at  $20\%$  FPR to show attack success, as it scores  $0\%$  at  $1\%$  FPR.
Metric →	Sim ↑	Detection Accuracy ↓
W.M.	D.GPT	O.AI
GPT2-1.5B	-	100.0	74.9	59.2
+ DIPPER 20L	99.5	98.9	45.7	35.3
+ DIPPER 40L	99.0	90.7	28.0	34.4
+ DIPPER 60L	97.5	71.1	15.8	31.3
+ 60L, 600	96.2	55.8	7.6	32.7
OPT-13B	-	100.0	29.8	33.5
+ DIPPER 20L	99.6	98.3	15.0	24.5
+ DIPPER 40L	99.4	87.3	6.4	24.1
+ DIPPER 60L	96.5	65.5	3.2	21.6
+ 60L, 600	92.9	51.4	1.5	21.6
GPT-3.5-175B
davinci-003	-	-	67.0*	40.5
+ DIPPER 20L	99.9	-	54.0*	43.1
+ DIPPER 40L	99.8	-	36.0*	43.1
+ DIPPER 60L	99.5	-	23.0*	40.1
+ 60L, 600	98.3	-	14.0*	38.1
Human Text	-	1.0	1.0	1.0
4.3 Attacking AI-generated text detectors
We present our results in Table 1 and Figure 3. Overall we find that:
Paraphrasing significantly lowers detection accuracy while preserving input semantics. Across all LMs, detectors, $^{11}$  and tasks, paraphrasing significantly lowers detection accuracy across all diversity control codes. For instance, paraphrasing GPT2-XL open-ended generations reduces watermark detection accuracy from  $100\%$  to  $57.2\%$ , and DetectGPT accuracy from  $70.3\%$  to just  $4.6\%$ . Trends are similar even for large LMs like GPT-3.5, for which paraphrasing reduces OpenAI's text classifier accuracy from  $30.0\%$  to  $15.6\%$ . Additionally, DIPPER preserves semantics effectively,