SlowGuess/NeurIPS_2023_OCR huggingface dataset:
${}^{6}$  Alternatively,if a random half of the human-written data was used for threshold adjustment,we find the other half has a FPR of  ${0.8}\%  - {1.2}\%$  across random splits,and this deviation will reduce with a bigger dataset.
<sup>7</sup>We consider both model-specific and model-agnostic detectors, as justified in Appendix F.3.
This classifier was taken down in July 2023 by OpenAI due to its low accuracy.
<sup>9</sup>While RankGen was not explicitly optimized for this task, it was trained to identify well-written continuations, so we hypothesize that it could also act as a reasonable AI-generated text detector.
<sup>10</sup>We discuss this attack briefly in Section 4.4.
Except RankGen, which scores paraphrases as AI-generated more often than non-paraphrased text. We attribute this to paraphrases being poorer continuations to the prompt compared to the original (Appendix C), an aspect RankGen bases its score on. However, it has low overall performance since it is not trained for this task.
as  $88\% - 99\%$  paraphrases achieve a P-SP score higher than the median score of human-written paraphrases. High semantic preservation is supported by careful human evaluations in Appendix C.2. Overall, we find that watermarking is the most resilient detector to paraphrasing.
Non-watermarking detectors are generally ineffective. We observe that all detectors apart from watermarking struggle with text generated by larger models like OPT-13B and GPT-3.5, achieving detection accuracies  $< 50\%$ . While DetectGPT is effective on the smaller GPT2-XL model (74.9% on long-form QA), its accuracy drops to just 29.8% on OPT-13B. Furthermore, GPTZero and RankGen perform the worst among the five detectors on all tested LMs (Table 1), as they are only able to detect  $< 15\%$  of non-paraphrased AI-generated text. Thus, we recommend against using these detectors.
ROC plots confirm the trends at different false positive rates. In Figure 4, we plot the detection accuracy (true positive rate) at different values of FPR between  $0\%$  and  $1\%$  for GPT2-XL. Overall, paraphrasing significantly drops detection rates across all FPR thresholds (more plots in Appendix H).
4.4 Alternative paraphrasing attacks
Paraphrasing multiple times: Our presented attacks use just a single paraphrase generated by DIPPER to
evade detection. A simple way to further improve the effectiveness of a paraphrase attack is to sample multiple times $^{12}$  from DIPPER and choose a paraphrase that evades the detector while also preserving semantics. We do not perform this attack as it can only be done if an attacker has access to a detector, which may be a strong assumption (see Appendix A.2). That being said, using multiple paraphrase samples can make the attacks even more potent against publicly available detectors.
Figure 4: ROC plots (0-1% FPR) for GPT2-XL using different detectors, before (solid lines) and after paraphrasing (dashed). Paraphrasing reduces detection rate across FPRs, and our detector retrieval detects paraphrases best; full plots in Appendix H.
Non-DIPPER paraphrasers: A second alternative is to use non-DIPPER paraphrasers that operate at the sentence level. These models can be deployed for long-form text inputs by paraphrasing the inputs sentence by sentence, ignoring prompt context. While the concurrent work of Sadasivan et al. [2023] shows that this method can also evade detection, our ablations in Appendix C show that non-contextual versions of DIPPER have lower quality and are less compatible with the prompt as DIPPER paraphrasers. Moreover, most existing paraphrasers lack fine-grained diversity control and multi-sentence input support (survey in Appendix D.1), two desired properties from an attacker's point of view: attackers want to modify long multi-sentence responses just enough to evade detection.
A more interesting option is to use an LLM like ChatGPT to perform few-shot contextual paraphrasing. While this method is likely to provide accurate paraphrases,[13] they may be detectable by strategies like watermarking (whether using the same API as the original LLM or a different one). We thus expect a sophisticated adversary to use their own private paraphraseer (like DIPPER) to evade detection.
5 Defense against paraphrase attacks using retrieval
In Section 4.3, we showed that paraphrasing is an effective attack against AI-generated text detectors. How can LLM API providers defend against these attacks? In this section, we propose retrieval over previously-generated sequences as a defense against paraphrase attacks. At a high level (Figure 5), an API provider first stores every sequence generated by their LLM in a database. The API provider offers an interface that allows users to enter candidate AI-generated text as a query. The interface searches
${}^{12}$  Precisely,compute  ${f}_{\text{dipper }}\left( x\right)$  for different random seeds while sampling text. Alternatively,an attacker could also compute  ${f}_{\text{dipper }}\left( {{f}_{\text{dipper }}\left( {\ldots {f}_{\text{dipper }}\left( x\right) }\right) }\right)$  ,but this will lead to excessive semantic drift from  $x$  .
<sup>13</sup>In initial experiments, we observed that DIPPER performs competitively with the much larger and more powerful GPT-3.5 davinci-003 model in terms of paraphrase quality, and significantly better at controlling diversity. This finding shows that specialized smaller models can outperform LLMs in paraphrasing tasks.
Figure 5: An illustration of AI-generated text detection with retrieval. Several users (including the attacker, shown as the purple emoji) feed prompts to the API which are collectively added to a private API-side database. Candidate queries are compared against this database using a retriever like BM25.
over the entire database of previously-generated text, trying to find a sequence that approximately matches the content of the input query. This search can be done using a semantic similarity scorer like P-SP [Wieting et al., 2022] or a retriever like BM25 [Robertson et al., 1995]. Since paraphrasing approximately preserves input semantics, we expect such a defense to still be able to map paraphrased generations to their source. We formalize our detector in Section 5.1, and then conduct a controlled comparison with competing detectors in Section 5.2. We evaluate retrieval-based detection at scale using a large retrieval corpus of 15M generations in Section 5.3. In Appendix A we extensively discuss limitations of retrieval-based detection and share ideas for enabling further scaling.
5.1 Formulating the retrieval defense
Let  $f_{\mathrm{LM}}$  be an LLM API (e.g., GPT-3.5) that takes a prompt  $x$  as input and returns a continuation  $y$ . Let  $f_{\mathrm{ret}}$  be an encoder (e.g., TF-IDF, neural network) that embeds variable-length sequences into fixed-size vectors that represent the input semantics. Then, we do the following:
Building the database: Let  $x_{1}, \ldots, x_{N}$  be the set of prompts that have been fed as input to the API in the past with  $y_{i} = f_{\mathrm{LM}}(x_{i})$  being the LLM output. Here  $N$  can potentially be very large for popular APIs (we study up to  $N = 15\mathrm{M}$ ). We construct our database  $\mathbf{Y} = [\mathbf{y}_1, \ldots, \mathbf{y}_N]$  by encoding every LLM API output with our retrieval encoder, or  $\mathbf{y}_{i} = f_{\mathrm{ret}}(y_{i})$ . The database  $\mathbf{Y}$  is dynamically updated and stored on the API side. It is inaccessible to clients except via the API described in the next step.
Querying the database: Let  $y'$  be a candidate text and  $\mathbf{y}' = f_{\mathrm{ret}}(y')$  its encoded vector. Suppose a client wishes to know whether  $y'$  was generated by the API  $f_{\mathrm{LM}}$ . The API provider can check this by seeing whether the maximum similarity score of  $y'$  to an entry in the database exceeds some detection threshold  $T$  chosen by the API provider:
$$
\text {o u t p u t} = \text {s c o r e} > T, \text {w h e r e s c o r e} = \max  _ {i \in \{1,.. N \}} \frac {\mathbf {y} ^ {\prime} \cdot \mathbf {y} _ {i}}{\left| \mathbf {y} ^ {\prime} \right| \left| \mathbf {y} _ {i} \right|}
$$
We expect unperturbed machine-generated text to always get a score of 1.0, while paraphrasing the text may lower the detection score. Hence, lowering  $T$  will increase the detection rate of heavily-paraphrased text but also increase the false positive rate (i.e., human-written text that resembles sequences previously generated by the LLM API can be falsely flagged). Since  $N$  can be very large, the score can also be approximated using efficient nearest neighbor libraries like FAISS [Johnson et al., 2019]. However, in this work we only compute exact inner products.
As the retriever  $f_{\mathrm{ret}}$ , we experiment with two choices: P-SP [Wieting et al., 2022] and BM25 [Robertson et al., 1995]. We implement BM25 using the retriv library from Bassani [2022]. In order to normalize and calibrate BM25 scores, we compute the F1-score unigram token overlap [Rajpurkar et al., 2016] between the candidate  $y'$  and the best retrieval  $y*$  to get a detection score in [0, 1].
5.2 Controlled comparisons of retrieval with other AI-generated text detectors
First, we conduct a controlled comparison between the detection algorithms evaluated in Section 4.3 and our retrieval method on long-form question answering. $^{14}$  We construct three kinds of databases,
1. 3K sequences generated by a specific LM for one of the tasks;
$^{14}$ Corresponding results in open-ended text generation on Wikipedia prompts are provided in Appendix G.1.
Table 2: A comparison of retrieval against other detectors on long-form QA (300 generated tokens). Our detector outperforms baselines (at  $1\%$  FPR) even with the most diverse paraphrases  $(+60\mathrm{L},\mathrm{O})$
GPT2-XL	OPT-13B	GPT-3.5 (davinci-003)
Original	+ 60L	+ 60 L,O	Original	+ 60L	+ 60 L,O	Original	+ 60L	+ 60 L,O
Watermark [2023a]	100.0	71.1	55.8	100.0	65.5	51.4	-	-	-
DetectGPT [2023]	74.9	15.8	7.6	29.8	3.2	1.5	1.0	0.0	0.0
OpenAI [2023a]	59.2	31.3	32.7	33.5	21.6	21.6	40.5	40.1	38.1
(Ours) Retrieval over corpus of 3K generations from model itself, with retriever:
SP	100.0	95.6	87.7	100.0	94.8	85.3	100.0	94.2	85.1
BM25	100.0	99.2	97.8	100.0	99.3	97.3	100.0	98.6	96.2
(Ours) Retrieval over corpus of 9K generations pooled from all three models, with retriever:
SP	100.0	88.9	75.4	100.0	89.6	76.4	100.0	93.8	84.6
BM25	100.0	98.3	95.2	100.0	98.5	94.4	100.0	98.5	96.0
(Ours) Retrieval over 43K ShareGPT responses + corpus of 3K generations from model itself, with retriever:
SP	100.0	94.0	84.8	100.0	94.2	84.7	100.0	94.1	84.9
BM25	100.0	98.9	97.5	100.0	99.0	97.3	100.0	98.4	95.5
2. 9K sequences formed by concatenating the generations from all three LMs in this paper;
3. 46K sequences constructed by combining the 3K sequences from (1) with 43K LLM responses from ShareGPT-Vicuna.[15]
We expect (2) to be a more difficult test for our method than (1), since the retriever needs to distinguish between multiple generations from different models given the same prompt. On the other hand, (3) denotes a more real-world setting with several diverse LLM-generated outputs from ShareGPT. Next, we perform retrieval over this corpus using different types of queries: the original AI-generated text, its DIPPER paraphrase, and human-written text (each query with at least 50 tokens).
Table 2 shows that across all LMs, retrieval is a much more effective detector than baseline detectors. On unperturbed AI-generated text, retrieval has a  $100\%$  detection accuracy due to exact match with the retrieval corpus. On paraphrased text, retrieval with BM25 is quite effective, detecting  $97.8\%$  of the highest-diversity paraphrases (L60, O60) on GPT2-XL,  $97.3\%$  on OPT-13B and  $96.2\%$  on GPT-3.5 in long-form question answering. This is significantly better than the next best alternative with competing detectors ( $55.8\%$ ,  $51.4\%$ ,  $38.1\%$ ). Even on our harder augmented databases, detection rates continue to be high:  $95.2\%$ ,  $94.4\%$ ,  $96.0\%$  for the 9K augmented database;  $97.5\%$ ,  $97.3\%$ ,  $95.5\%$  for the ShareGPT augmented database. Finally, we observe that BM25 is a more effective retriever than P-SP, scoring  $95.2\%$  vs  $75.4\%$  on the augmented setting in GPT2-XL. These trends are consistent across different FPR thresholds, as shown in Figure 4.
In Appendix G.2, we additionally observe promising preliminary results that show the effectiveness of retrieval against text mixing attacks [Kirchenbauer et al., 2023b].
5.3 Is retrieval an effective detector with a large retrieval corpus?
In the previous section, we conducted experiments using the set of 9K sequences generated by all three models as the retrieval corpus. However, this is more of a toy experiment: in practice, a popular LLM API may serve millions of queries a day. As the corpus grows larger, the false positive rate (i.e., human-written text falsely detected as AI-generated) will grow. How well do retrieval-based detectors scale? To answer this question, we need access to a large corpus of AI-generated text. We utilize the training data used to train RankGen [Krishna et al., 2022a], which contains over 70M AI-generated sequences. We use the Project Gutenberg and Wikipedia splits of the training data, each of which contain 15M sequences generated by a T5-XXL model [Raffel et al., 2020] fine-tuned on the different documents in the same domain. We discard generations which are shorter than 50 tokens, and paraphrase a subset of 2K generations to evaluate retrieval.
Retrieval is effective even with a corpus size of 15M generations. In Figure 6a, we plot the detection accuracy as a function of retrieval database size. Overall, we observe that detection
15huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered
Figure 6: Detection rate using retrieval at  $1\%$  FPR w.r.t. corpus size (left) and query length (right).
accuracy remains consistently high across different corpus sizes (varying from 1M generations to 15M generations). We observe slight drops in performance as the corpus size increases: just  $1\%$  (98.3 to 97.3) on Project Gutenberg (PG19) and  $9.6\%$  (90.0 to 80.4) on Wikipedia. Consistent with the results in Section 5.2, BM25 continues to outperform P-SP across different corpus sizes.
Retrieval detection works best with 50 or more tokens of generated text. Another important factor for our retrieval-based detector is the query length: shorter queries are likely to have more matches (many of them spurious) compared to longer ones. In Figure 6b, we plot the detection accuracy of paraphrased sequences at various query lengths by truncating each sequence to its first  $X$  words before using it as a query for BM25. We use a retrieval corpus of 2M generations for this experiment. We observe that BM25 struggles to detect paraphrased text with a query length of 20 (less than  $25\%$  accuracy), but the detection rate rapidly increases and begins to plateau at 50 tokens.
5.4 Scalability and limitations of retrieval-based detectors
In Appendix A we extensively discuss the scalability of retrieval (A.1), its limitations (A.2), ideas for improving retrieval-based detectors (A.3), and incentive structures for LLM providers to implement retrieval (A.4). In summary, we believe retrieval-based detection is a scalable approach: we estimate that if OpenAI implemented it with ChatGPT, they would need just 5TB of storage space per month (similar to modern portable hard disks). Furthermore, retrieval on ChatGPT scale takes 130 seconds per retrieval on a CPU-only MacBook Pro, which can certainly be further optimized. However, retrieval-based detection has some important limitations: (1) potential privacy risk of exposing all LLM responses behind a binary classifier; (2) inability to use retrieval-based detection on open-source LLMs like LLAMA [Touvron et al., 2023]; and (3) the need to implement and maintain retrieval infrastructure. We discuss mitigation strategies for the limitations in Appendix A.2.
We present DIPPER, a controllable paraphraseer that can rewrite paragraphs in context. We use DIPPER to stress test current AI-generated text detectors, and we find that DIPPER paraphrases easily evade these detectors while preserving input semantics. As a defense, we propose a simple retrieval-based detector which searches through a corpus of previously-generated sequences from an LLM API for semantically-similar generations to a given query. We show that this defense significantly outperforms baselines on paraphrased text, and scales effectively. We discuss the limitations and ethical considerations of our work in Appendix A, B. We have additionally open sourced our models, code and data to enable future research. Since this paper's initial release, DIPPER has been extensively utilized in follow-up studies to measure the robustness of AI-generated text detection algorithms ([Lu et al., 2023, Koike et al., 2023, Zhao et al., 2023, Yoo et al., 2023, Patil et al., 2023, Kirchenbauer et al., 2023b, Kumarage et al., 2023, Liu et al., 2023], to name a few).
We are very grateful to our anonymous reviewers for their detailed feedback which helped improve our paper. We also thank Kenton Lee, Katherine Thai, Slav Petrov, Fernando Pereira, Jon Clark, William Cohen, Tu Vu, Yapei Chang, Shufan Wang and the UMass NLP group for several useful discussions on earlier versions of this paper. Kalpesh Krishna was supported by a Google PhD Fellowship. Part of this work was done while Kalpesh was a Student Researcher at Google Research, hosted by John Wieting.
S. Abdelnabi and M. Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pages 121-140, 2021.
E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016. doi: 10.18653/v1/S16-1081. URL https://www.aclweb.org/anthology/S16-1081.
R. R. Agrawal, M. Turchi, and M. Negri. Contextual handling in neural machine translation: Look behind, ahead and on both sides. In 21st annual conference of the European association for machine translation, pages 11-20, 2018.
S. Badaskar, S. Agarwal, and S. Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II, 2008. URL https://aclanthology.org/I08-2115.
A. Bakhtin, S. Gross, M. Ott, Y. Deng, M. Ranzato, and A. Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019.
E. Bandel, R. Aharonov, M. Shmueli-Scheuer, I. Shnayderman, N. Slonim, and L. Ein-Dor. Quality controlled paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 596-609, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.45. URL https://aclanthology.org/2022.acl-long.45.
E. Bassani. retriv: A user-friendly and efficient search engine in python., 2022. URL https://github.com/AmenRa/retriv.
D. Beresneva. Computer-generated text detection using machine learning: A systematic review. In 21st International Conference on Applications of Natural Language to Information Systems, NLDB, pages 421-426. Springer, 2016.
R. Bhagat and E. Hovy. Squibs: What is a paraphrase? Computational Linguistics, 39(3):463-472, 2013. doi: 10.1162/COLI_a_00166. URL https://www.aclweb.org/anthology/J13-3001.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
N. Carlini, F. Tramér, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ü. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650. USENIX Association, Aug. 2021. ISBN 978-1-939133-24-3. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting.
J. Wieting, K. Gimpel, G. Neubig, and T. Berg-kirkpatrick. Paraphrastic representations at scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 379-388, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-demos.38.
S. Witteveen and M. Andrews. Paraphrasing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 215-220, Hong Kong, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5623. URL https://aclanthology.org/D19-5623.
X. Xie, X. Lu, and B. Chen. Multi-task learning for paraphrase generation with keyword and part-of-speech reconstruction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1234-1243, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.97. URL https://aclanthology.org/2022.findings-acl.97.
H. Xiong, Z. He, H. Wu, and H. Wang. Modeling coherence for discourse neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7338-7345, 2019.
W. Xu, C. Callison-Burch, and C. Naples. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283-297, 2015.
K. Yang, D. Liu, W. Lei, B. Yang, H. Zhang, X. Zhao, W. Yao, and B. Chen. Gcpg: A general framework for controllable paraphrase generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4035-4047, 2022.
K. Yin, P. Fernandes, A. F. Martins, and G. Neubig. When does translation require context? a data-driven, multilingual exploration. arXiv preprint arXiv:2109.07446, 2021.
K. Yoo, W. Ahn, and N. Kwak. Advancing beyond identification: Multi-bit watermark for language models. arXiv preprint arXiv:2308.00221, 2023.
R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
J. Zhang, H. Luan, M. Sun, F. Zhai, J. Xu, M. Zhang, and Y. Liu. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v1/D18-1049. URL https://aclanthology.org/D18-1049.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
X. Zhao, P. Ananth, L. Li, and Y.-X. Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439, 2023.
A Limitations of retrieval-based detection and ideas for scaling it further