SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This section extensively discusses the scalability, limitations, future work and and an LLM API provider's incentive structures for retrieval-based detection. First, in Appendix A.1, we discuss the scalability of retrieval-based detection in terms of compute requirements, storage space, and accuracy on larger databases. Next, in Appendix A.2, we first point out some limitations of using retrieval for AI-generated text detection (Section 5), some of which potentially apply to all existing detectors. Along with limitations, we provide several possible workarounds. In Appendix A.3, we then discuss ideas that can make the proposed retrieval detection work well at an even larger scale than the one we discussed in Section 5. Finally, in Appendix A.4 we briefly discuss why an LLM provider may be incentivized to implement retrieval-based detection, and the relationship of this detector with GDPR's right to be forgotten.
A.1 Scalability of retrieval-based detection
Retrieval-based detection requires the storage of a large database of LLM-generated responses, and querying this database to find matches for previously generated responses. How scalable is this approach in terms of storage space, compute, and accuracy? In this section we perform approximate calculations of these requirements on OpenAI's ChatGPT [Schulman et al., 2022].
Storage space requirements: We estimate ChatGPT's outputs to take 5TB space monthly (similar to a personal portable hard-disk) via the following calculations. ChatGPT currently gets about 2B monthly visits [NewYorkPost, 2023]. Assuming an average response length of 500 tokens per session, this corresponds to 1 trillion tokens. Similar in size to LLaMA's training data [Touvron et al., 2023], this needs 5TB space. However, 5TB is a small amount of storage compared to the industrial scale of information retrieval. For example, the Google Search index is over 100,000TB and has 100B+ pages [Google, 2023]. Major LLM service providers already have complex storage infrastructure to facilitate this defense. Additionally, OpenAI already stores conversations for at least 30 days (to monitor abuse, and potentially RLHF), even after a user chooses to opt-out [OpenAI, 2023b].
Compute requirements: Our retrieval experiments, conducted on a 14-core CPU (similar to a Macbook Pro), took 1 second per retrieval on a 15M sized corpus. Extrapolating to a corpus of ChatGPT's monthly usage (2B visits) would need 130 seconds/retrieval on a Macbook. However, this is fully parallelizable, and can make use of GPUs (Google searches  $100\mathrm{B}+$  entries in  $< 1$  sec). Moreover, efficient similarity search has powerful libraries like FAISS available [Johnson et al., 2019]. For comparison, ChatGPT itself takes 10 seconds/response, possibly using a powerful 8-GPU A100 server. $^{16}$  Major LLM providers have massive compute clusters, and we believe the computational requirement of retrieval is much lower than hosting LLMs in the first place, which these providers are already adept at. Moreover, our proposed ideas in Appendix A.3 can further reduce compute costs.
Accuracy on larger databases: Our experiments were conducted on the RankGen training set [Krishna et al., 2022a], which is the largest publicly available database of AI-generated text that we are aware of (15M generations each in four domains). Besides this, in Section 5 we also conducted experiments on a ShareGPT corpus with 47K ChatGPT-generated responses. We note that it is extremely expensive and time consuming to create a corpus of AI-generated text from scratch: at a cost of  $0.001 per 500-word response, collecting a billion ChatGPT outputs would cost$ 1M and take a long time to collect due to rate limits. Hence, a billion-scale experiment is likely only possible with an LLM provider's private database.
One of the concerns with a larger database is that of semantic collisions: a database will saturate with entries having similar semantics, especially for popular topics, and subsequently harm detection at scale. However, we note that:
1. Like other detectors, retrieval works best on longer sequences (Figure 6b). Long generations exponentially increase the likelihood of semantic divergences between pairs of entries.
$^{16}$ https://twitter.com/tomgoldsteincs/status/1600196981955100694
2. Retrieval compares the candidate response against the top-1 entry in the database, not the top-k. For false-positive candidates on popular topics, the top-k entries together are more likely to cover input semantics (recall) rather than top-1 (precision).
3. The most effective retrievers use a combination of neural semantic encoders and token overlap scores [Thakur et al., 2021]. We also see this our experiments (Section 5), BM25 beats P-SP at detection. BM25 is not purely semantic driven: it uses TF-IDF token overlap.
4. The retrieval accuracy for unperturbed AI-generated text is always  $100\%$ , just like exact match searches in a modern search engine. Retrieval-based detection is also effective on substrings of unperturbed text (as shown in Appendix G.2).
Overall, we are optimistic about our scaling plots (Figure 6a), and see just a  $0.8\%$  drop moving from a 1M to 10M database (PG19-BM25). We emphasize that BM25 is a basic retriever, and is not optimized on our task. Information retrieval literature has many powerful retrievers and has shown success at billion-scale corpus sizes [Gomes et al., 2013, Lakshman et al., 2021]. Google Search currently effectively indexes over 100B webpages [Google, 2023]. We have also suggested a dense retrieval mechanism in Appendix A.3 which can be optimized on the underlying retrieval corpus.
Retrieval can easily be used in tandem with other detectors like watermarking. Finally, we note that our paper is the first proof-of-concept that shows a retrieval-based detector could work, and we anticipate future work to build upon it.
A.2 Limitations of retrieval for detection
While retrieval over previously-generated sequences is an effective defense against paraphrase attacks, it also suffers from key limitations, some of which apply broadly to all existing detectors. We discuss these limitations below and discuss possible solutions:
1. Detection is specific to an API. Unlike other general-purpose AI detection algorithms e.g. OpenAI's classifier [OpenAI, 2023a], retrieval can only detect generations from the API over which the database is built. API #1 has no access to the database of generations from API #2, and thus will not be able to detect generations produced by API #2.
2. Retrieval is limited to closed-source LLMs. Users of open-source LLMs like LLAMA [Touvron et al., 2023] can freely generate outputs without the outputs being stored in a central database. However, currently most major LLM providers operate their LLMs behind closed APIs. It is also important to note that watermarking [Kirchenbauer et al., 2023a], the most promising alternative to retrieval, also has this limitation. Since watermarks are added during decoding rather than into the model weights, users of open LLMs are free to generate text without watermarks. While other alternatives like DetectGPT [Mitchell et al., 2023] or classifiers do not suffer from this issue, we show that they either have low accuracy, or are extremely vulnerable to paraphrasing (Section 4.3).
3. The API provider needs to provide a retrieval infrastructure. After the release of ChatGPT [Schulman et al., 2022], AI chatbots are getting widespread adoption. At a conservative rate of 5M queries a day, the database will have almost two billion entries in a year. Complex retrieval infrastructure (like modern search engines) will be necessary to retrieve over these large databases with low latency.
4. False positives due to training data memorization. Language models have been shown to memorize sequences verbatim from their training data [Carlini et al., 2021], such as the Gettysburg Address [Radford et al., 2019]. Despite being originally written by humans, these sequences will be classified as model-generated by our detector. To tackle this issue, we suggest API providers additionally perform retrieval over the training data used to train the model. If a sequence is found in the training set as well as the generation database, it is likely to be an instance of training set memorization.
5. Privacy concerns. Providing a retrieval detection service partially exposes the database of previously generated text for all users. This raises concerns of membership inference attacks [Shokri et al., 2017] on private user data which may appear in the generated text (if present in user prompt). To mitigate this, we suggest: (1) the detection service should be provided only to trusted users under an agreement to not misuse the system, such as college teachers trying to detect cheating; (2) users should be encouraged not to enter any sensitive private data in their prompts to APIs, a
practice already followed by ChatGPT $^{17}$ ; (3) API providers only provide a binary output from this detector (AI-generated or not), rather than actual search results; (4) API providers rate-limit queries from IP addresses; and (5) differential privacy mechanisms or scrubbing private attributes to make it difficult / impossible to reconstruct the user prompt from just detector access.
6. Slight reduction in accuracy with large databases. As we observed in Section 5.3, the accuracy of detecting paraphrased text slightly degrades as the database of retrievals gets larger. However, we found this decrease to be quite small (only  $1\%$  on PG19 scaling 1M generations to 15M), despite using fairly primitive retrievers like BM25. Moreover, unperturbed AI-generated text will always be detected with  $100\%$  accuracy using our method, irrespective of corpus size. We discuss this topic more in Appendix A.1 under "Accuracy on larger databases".
7. Tasks with constrained output space or short outputs. Similar to all other detection algorithms, it may be hard or even impossible to distinguish AI-generated outputs for tasks with a constrained output space (like sentence-level translation, classification) or very short outputs (as shown in Section 5.3). Thus, we believe the main utility of AI-generated text detection is for longer-form generated text, and hence we focus on tasks like long-form QA and open-ended text generation with relatively lengthy outputs. Note that to avoid detection, a sophisticated attacker may try to generate long-form text in smaller chunks using multiple API calls, where each newly-generated chunk is incrementally concatenated to the prompt. This is not a concern for our method if retrieval is done over the corpus of prompts concatenated with generations.
8. Iterative attacks with access to detector. Another concern is that attackers with access to detection algorithms will iteratively modify their perturbations until they avoid detection (as shown by Sadasivan et al. [2023]). While this is a valid concern for all detectors, we believe retrieval has an important advantage over the alternatives. Since the corpus of previously-generated text is proprietary, only the API provider can provide access to this detection service - it is impossible for attackers to locally reproduce this detector. This allows API providers to adopt several mitigation strategies such as (1) rate-limiting queries to avoid iterative attacks; (2) providing retrieval access only to verified users (e.g., teachers); and (3) detecting possible iterative attacks by analyzing previously queries to the retriever.
9. Lack of formal guarantees between threshold  $T$ , response length and detection rate. Unlike watermarking, we believe it is harder to establish formal relationships between the chosen threshold  $T$ , response length, and the corresponding TPR / FPR rates. Similar to DetectGPT / classifiers, we believe the threshold needs to be estimated empirically on the underlying corpus, retriever and candidate response distribution. A formal relationship between FPR and the threshold (like in watermarking) may be possible using information about the density of the retrieval database in the semantic vector space. We leave this exploration for future work.
A.3 Ideas to make retrieval detection work well at an even larger scale
In Section 5.3, we observed that our proposed retrieval detector is effective even with a large corpus of 15M previously-generated sequences. While we do not have access to a larger corpus of generations (billion-scale), in this section we describe some ideas to improve retrieval detection at such a scale.
1. Timestamp filtering in retrieval corpus. To reduce the large search space, the detector interface could provide users with an option to restrict retrieval to only a fixed time period during which the text was likely to be generated. For instance, a common use-case of AI-generated text detection might be when teachers attempt to catch plagiarism in college essays. Teachers could restrict retrieval to only those generations created during the assignment window.
2. More sophisticated retrieval strategies. In our work, we only explore simple retrieval strategies like BM25. However, several more sophisticated retrieval strategies exist, which are known to boost performance [Thakur et al., 2021] and could be useful here. These include methods like re-ranking of top- $k$  retrievals [Khattab and Zaharia, 2020] or dense retrieval [Karpukhin et al., 2020]. We do note that these more complex methods are also slower, and latency is likely to be a pressing concern for API providers.
3. Fine-tuning dense retrievers for the detection task. The retrievers in our work are not fine-tuned for the task of AI-generated text detection. However, we hypothesize that fine-tuning
$^{17}$ https://chat.openai.com
retrievers on this task can help retrievers adapt better to the retrieval corpus and detection task. Specifically, a contrastive learning approach could be adopted here: positive pairs are paraphrased or otherwise noised sequences paired with their generations, while negative pairs are human-written continuations paired with the machine-generated text.
A.4 Incentives for LLM providers to implement retrieval-based detection
Finally, in this section we discuss the incentive structures for LLM API providers, and why they might want to implement retrieval-based detection.
1. There is a substantial push from both the US and European governments to regulate companies to make their AI-generated text/images detectable. For instance, several major LLM providers made voluntary commitments to watermark their AI-generated content [Reuters, 2023a]. Similarly, the European government recently pushed for rulings to make AI-generated content detectable [Reuters, 2023b].
2. In a hypothetical scenario where there is a lawsuit about the origin of some malicious AI-generated content, maintaining a database of previously generated responses could be a reliable method to prove innocence, given its strong performance over competing AI-generated text detectors. Keeping this dataset private, also protects LLM providers from the privacy risks of retrieval-based detection (Appendix A.2).
3. Major LLM providers are already storing their model-generated outputs to monitor abuse, as well as possibly help improve their products with RLHF preference-based training. For instance, OpenAI's ChatGPT stores the history for 1 month even if users choose to opt-out [OpenAI, 2023b]. Hence, implementing a retrieval-based detection service on top of this database does not entail a resource overhead in terms of storage, and reduces the engineering effort needed to implement this detector.
Is retrieval-based detection at odds with GDPR's right-to-be-forgotten? One of the hurdles for LLM providers to implement retrieval-based detection is GDPR's right to be forgotten,[18] which allows attackers to request their data to be deleted by the LLM provider in order to avoid detection. However, we believe that AI-generated text detection is a sufficient cause to temporarily override GDPR deletion requests. We believe that AI-generated text detection could fall under the following GDPR guidelines for overrides: (1) "freedom of expression and information", (2) "establishment of a legal defense or in the exercise of other legal claims", and possibly (3) "comply with a legal ruling or obligation" in the future. As an example of this, while OpenAI allows users to delete their chat history [OpenAI, 2023b], they retain it for 30 days, and can review it if required to monitor for abuse.
B Ethical Considerations
Our goal in this paper is not to provide a recipe for potential attackers (e.g., college students wishing to use ChatGPT in their essays) to evade AI text detection systems. Rather, we wish to bring awareness to the wider community about the vulnerabilities of current AI-generated text detectors to simple paraphrase attacks. These detectors are not useful in their current state given how easy they are to evade. We encourage the research community to stress test their detectors against paraphrases, and to develop new detectors which are robust against these attacks. To facilitate such research, we open source our paraphraseer and associated data / code.
Furthermore, we propose not just an attack but also a potentially strong defense against this attack. Our detection strategy is simple, relying on retrieval over a corpus of previously-generated sequences. We empirically show that such a detection algorithm could work at scale and provide extensive discussion on possible methods to improve performance (Appendix A.3), as well as discussing possible limitations and approaches to tackling them (Appendix A.2). We hope that retrieval-based AI-generated text detectors rapidly improve and are eventually deployed in conjunction with other detection methods like watermarking / classifiers.
18. https://gdpr.eu/right-to-be-forgotten
C Experiments measuring intrinsic paraphrase generation quality
Our experiments in Section 4 and Section 5 focused on attacking AI-generated text detectors with paraphrases and defending against these paraphrase attacks. We used DIPPER as the underlying paraphrase generation model for all of these experiments. Are DIPPER's paraphrases actually good enough to make the attack worthwhile, and can simpler paraphrasers be just as effective as DIPPER? In this section, we conduct careful ablation experiments (Appendix C.1) and human evaluations (Appendix C.2) to validate the effectiveness of DIPPER at preserving the semantics of the input generation. Our results show that DIPPER effectively leverages surrounding context to paraphrase multiple sentences while preserving input semantics.
C.1 Ablation studies on DIPPER
In this section, we perform automatic evaluations to confirm the efficacy of DIPPER as a paraphraseer. From a survey of existing paraphraseers that we carry out in Appendix D.1, DIPPER possess two unique features that differentiate it from other paraphraseers: (1) its ability to leverage context from outside of the text to be paraphrased (such as the prompt); and (2) its ability to paraphrase multiple sentences at once. How useful are these features while paraphrasing long sequences of text?
To answer this question, we first train an ablated version of DIPPER by constructing a training dataset (Section 3) without any left or right context, and then fine-tuning T5-XXL using the same hyperparameters as in Section 3. We call this model DIPPER-no-ctx. We paraphrase 1K open-ended generations from GPT2-XL using both DIPPER and DIPPER-no-ctx, using each of the four configurations of diversity control codes studied in this paper. We then evaluate the quality of the paraphrased text using three metrics: (1) GPT3.5-davinci-003 perplexity [Brown et al., 2020] of the prompt concatenated with the paraphrased continuation; (2) RANKGEN compatibility between the prompt and the paraphrased continuation [Krishna et al., 2022a]; and (3) unigram token overlap between the paraphrased continuation and the prompt.
Contextual paraphrasing leads to higher quality paraphrases. In Table 3 (Experiment 1), we observe that across all four control code configurations and all three metrics, paraphrases from DIPPER are preferred over paraphrases from DIPPER-no-ctx. Specifically, with the lexical and order control codes set to  $60\%$  (most diverse), DIPPER paraphrases are preferred by GPT3.5 perplexity  $71\%$  of the time compared to non-contextual paraphrases (average perplexity drop of 12.9 vs 14.2).
Paraphrasing multiple sentences at a time is better than paraphrasing individual sentences. Next, we use our DIPPER-no-ctx model to compare two settings: paraphrasing 3 sentences at a time vs paraphrasing 1 sentence at a time before concatenating. We hypothesize that the former will produce higher quality paraphrases since we expect it to better connect discourse elements across the text. Indeed, in Table 3 (Experiment 2) across all control codes, GPT3.5 and RANKGEN usually prefer multi-sentence paraphrases over the single-sentence baseline. This preference is  $71\%$  or higher for all control codes when evaluating with GPT-3.5 perplexity, reaching  $83\%$  for L60,O60.
DIPPER paraphrases are close to the unperturbed GPT-2 XL generations. Finally, we compare DIPPER with the original GPT2-XL generations (without paraphrasing) on the same three metrics. While we expect metrics to prefer non-paraphrased text, a strong paraphrase will produce text that is close to the original in terms of these metrics. Table 3 (Experiment 3) confirms our hypothesis: at L20, RANKGEN has a 50-50 preference between the two outputs, while GPT3.5 prefers the non-paraphrased generations just  $61\%$  of the time, with an average perplexity gain of just 0.4 (11.1 to 11.5). At more diverse control codes, preference for GPT2-XL generations does go up (58% RANKGEN, 73% GPT3.5 for L60), but absolute scores continue to be close (11.1 vs 12.3 GPT-3.5 perplexity). Note that while all of these ablations use just a single paraphrase sample, it is easy for an attacker to obtain multiple samples from DIPPER and choose the sample that maximizes these metrics (as discussed in Section 4.3).
C.2 Human evaluation of semantic preservation using DIPPER
The automatic semantic similarity scores in Table 1 and 3 indicate that DIPPER generates paraphrases that are faithful to the original input paragraphs. To confirm this result with human evaluation, we
Table 3: Ablation experiments demonstrate the high quality of DIPPER's paraphrases compared to alternatives. Displayed scores are the percentage of cases in which rewrite A is preferred over B by one of the three metrics, with subscripts showing absolute average scores on each metric across the dataset. Overall, DIPPER benefits from context outside the input (Experiment 1), multi-sentence paraphrasing (Experiment 2), and is not too far behind non-paraphrased text in terms of quality (Experiment 3).
Open-ended generation with GPT2-XL on Wikipedia prompts
Control	RANKGEN-XL	GPT3.5 davinci-003 perplexity	unigram overlap with prompt
rewrite A	rewrite B	rewrite A	rewrite B	rewrite A	rewrite B
Experiment 1: Is context helpful for paraphrasing?
rewrite A = DIPPER with context
rewrite B = DIPPER no context
20L	62% 10.2	38% 9.4	58% 11.5	42% 11.7	55% 41.3	45% 40.7
40L	62% 9.8	38% 8.7	64% 11.9	36% 12.5	57% 40.7	43% 39.8
60L	64% 9.6	36% 7.9	66% 12.3	34% 13.3	55% 39.9	45% 39.2
60L,60O	66% 8.3	34% 6.3	71% 12.9	29% 14.2	56% 39.4	44% 38.4
Experiment 2: Is it helpful to paraphrase multiple sentences at a time?
rewrite A = DIPPER 3 sentences at a time
rewrite B = DIPPER 1 sentence at a time
20L	55% 9.4	45% 8.9	73% 11.7	27% 12.5	51% 40.7	49% 40.7
40L	55% 8.7	45% 8.5	71% 12.5	29% 13.3	46% 39.8	54% 40.3
60L	53% 7.9	47% 7.6	71% 13.3	29% 14.3	46% 39.2	54% 39.8
60L,60O	57% 6.3	43% 5.3	83% 14.2	17% 16.7	47% 38.4	53% 38.9
Experiment 3: Does paraphrasing preserve the quality of the original text?
rewrite A = no paraphrasing
rewrite B = DIPPER
20L	50% 10.4	50% 10.2	61% 11.1	39% 11.5	51% 41.6	49% 41.3
40L	57% 10.4	43% 9.8	67% 11.1	33% 11.9	55% 41.6	45% 40.7
60L	58% 10.4	42% 9.6	73% 11.1	27% 12.3	58% 41.6	42% 39.9
60L,60O	68% 10.4	32% 8.3	79% 11.1	21% 12.9	61% 41.6	39% 39.4