SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaomeng Hu
Sha Tin, Hong Kong
xmhu23@cse.cuhk.edu.hk
pin-yu.chen@ibm.com
Tsung-Yi Ho
tyho@cse.cuhk.edu.hk
Recent advances in large language models (LLMs) and the intensifying popularity of ChatGPT-like applications have blurred the boundary of high-quality text generation between humans and machines. However, in addition to the anticipated revolutionary changes to our technology and society, the difficulty of distinguishing LLM-generated texts (AI-text) from human-generated texts poses new challenges of misuse and fairness, such as fake content generation, plagiarism, and false accusations of innocent writers. While existing works show that current AI-text detectors are not robust to LLM-based paraphrasing, this paper aims to bridge this gap by proposing a new framework called RADAR, which jointly trains a robust AI-text detector via adversarial learning. RADAR is based on adversarial training of a paraphraseer and a detector. The paraphraseer's goal is to generate realistic content to evade AI-text detection. RADAR uses the feedback from the detector to update the paraphraseer, and vice versa. Evaluated with 8 different LLMs (Pythia, Dolly 2.0, Palmyra, Camel, GPT-J, Dolly 1.0, LLaMA, and Vicuna) across 4 datasets, experimental results show that RADAR significantly outperforms existing AI-text detection methods, especially when paraphrasing is in place. We also identify the strong transferability of RADAR from instruction-tuned LLMs to other LLMs, and evaluate the improved capability of RADAR via GPT-3.5-Turbo.
Project Page and Demos: https://radar.vizhub.ai
IBM demo is developed by Hendrik Strobelt and Benjamin Hoover at IBM Research HuggingFace demo is developed by Xiaomeng Hu
Large language models (LLMs) are high-capacity neural networks that are pretrained at web-scale datasets. They are foundation models achieving state-of-the-art performance in a wide range of natural language processing tasks (e.g. document completion, question answering, machine translation, and content creation with text prompts) with advanced capabilities such as in-context learning and reasoning (e.g. chain of thoughts). In particular, LLMs are the backbone of many ChatGPT-like conversational bots that enable text generation with high fluency and accuracy. However, while LLMs and their derived applications are expected to become ubiquitous in our future technology and society,
Figure 1: Overview of RADAR. An AI-text corpus is first generated from a target (frozen) language model from a human-text corpus. In RADAR, we introduce a paraphraseer (a tunable language model) and a detector (a separate tunable language model). In the training stage, the detector aims to discern human-text v.s. AI-text, while the paraphraseer aims to rewrite AI-text to evade detection. The model parameters of the paraphraseer and the detector are updated in an adversarial learning manner as described in Section 3. In the evaluation stage, the trained detector is deployed to predict the likelihood of AI-generated content for any input instance.
new risks in failing to distinguish the so-called "AI text" generated by LLMs have emerged and gained considerable attention due to various reasons. The problem of reliable AI-text detection is motivated by realistic socio-technological challenges such as fake content generation, AI plagiarism (e.g. using LLMs for writing tests), and false accusations of innocent writers. According to a report released by OpenAI<sup>1</sup>, their latest AI-text detector is admittedly not fully reliable. In the reported evaluation of some challenging cases for English texts, their classifier only correctly identifies  $26\%$  of AI-text (true positives) while incorrectly classifying  $9\%$  of human-written text (false positives). Moreover, a recent study [21] found that state-of-the-art AI-text detectors demonstrated severely degraded performance when encountering texts written by non-native English speakers.
What can be even more challenging in AI-text detection is that existing AI-text detectors are prone to be manipulated. The authors in [27, 17] showed that using LLMs as a paraphraseer can easily evade several AI-text detection methods, even in the scenario when the original AI-text had been watermarked. These findings sparked a heated debate about whether and how we can successfully design a reliable AI-text detector. While [27] theoretically quantifies the best detector's performance with respect to the total variation distance between AI-text and human-text distributions and argues that AI-text is difficult to detect, another work [3] proves that it is possible to obtain a reliable AI-text detector unless the human-text distribution is exactly the same as the AI-text distribution, based on an information-theoretical analysis (i.e., the sample complexity of Chernoff information and likelihood-ratio-based detectors).
To improve AI-text detection, we propose RADAR, a framework for training a robust AI-text detector using adversarial learning. An overview of RADAR is illustrated in Figure 1. Our proposal draws inspiration from adversarial machine learning techniques that train a high-quality generator by introducing a discriminator to form a two-player game, such as generative adversarial networks (GANs) [10]. In RADAR, we introduce a paraphraseer and a detector as two players with opposite objectives. The paraphraseer's goal is to generate realistic content that can evade AI-text detection, while the detector's goal is to enhance AI-text detectability. In our framework, both the paraphraseer and the detector are parametrized by separate LLMs. During training, the paraphraseer learns to rewrite the text from a training corpus (generated by a target LLM from a human-text corpus) with the
<https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text>
Figure 2: Performance evaluation (AUROC) of 8 LLMs over 4 human-text datasets. w/o paraphrase means the evaluation with the original AI-text corpora (the yellow bin  $\mathcal{M}$  in Figure 1). RADAR-Unseen paraphrase means the evaluation with the paraphrased AI-text (the green bin  $\mathcal{P}$  in Figure 1) generated from an independent paraphrase (OpenAI's GPT-3.5-Turbo API) that is not used in RADAR. The black error bar represents the standard deviation of the detection AUROCs across 8 LLMs. Please refer to Section 4.2 for more implementation details.
aim of decreasing the likelihood of AI-text prediction by the detector, whereas the detector aims to enhance the detection performance by learning to compare human-text vs.s. AI-text from the training data and the paraphraseer's output. These two players iteratively update their model parameters until their respective validation loss becomes stable. Specifically, the paraphraseer treats the prediction of the detector as a reward and uses Proximal Policy Optimization (PPO) [28] for updates. The detector updates its parameters based on a logistic loss function evaluated on the human-text and AI-text corpora (including the texts generated by the paraphraseer). In the evaluation phase, the trained detector is deployed to predict the likelihood of AI-written content for any input instance. When compared with 6 existing detectors, our experimental results on 8 different LLMs and 4 datasets show that RADAR attains similar detection performance on the original AI-generated texts (a relatively easy task) and simultaneously improves the AI-text detectability when facing an "unseen" paraphraseer (i.e. this paraphraseer is not used in RADAR). The result is summarized in Figure 2. When facing an unseen paraphraseer (GPT-3.5-Turbo), the area under the receiver operating characteristic (AUROC) score of RADAR is improved by  $31.64\%$  compared to the best existing detector, suggesting a significant improvement and reliable AI-text detection power enabled by RADAR.
- To the best of our knowledge, RADAR is the first study that leverages the idea of adversarial learning between a paraphraseer and a detector for training a robust AI-text detector.
- The experiments on 8 different LLMs (Pythia, Dolly 2.0, Palmyra, Camel, GPT-J, Dolly 1.0, LLaMA, and Vicuna) and 4 datasets show that unlike the six existing supervised and unsupervised AI-text detection methods, RADAR is the only robust detector that attains consistently high detection performance. RADAR's detector is not weakened by paraphrasing, as shown in Figure 2.
- We also find the strong transferability of RADAR's detection capability. The detectors of RADAR obtained from instruction-tuned first-class LLMs (e.g., Vicuna-7B) are also effective on other LLMs, suggesting the possibility of training a universal AI-text detector based on the state-of-the-art LLMs.
AI-Text Detection. The research in AI-text detection can be divided into three approaches. (i) Statistical methods: some statistics such as entropy [19], n-gram frequency, and perplexity are used as a threshold to discern AI-text. A typical example is GLTR [8], which exploits entropy, probability, and probability rank for detection. A more recent work is DetectGPT [23], which assumes that the machine-generated text always lies in the negative curvature region of the log probability of the LLM of interest. Based on this hypothesis, DetectGPT perturbs the input text with a mask-filling language
model, such as T5 [25]. Then, AI-text detection is performed by comparing the log probability of the text and its infilled variants. (ii) Classification methods: AI-text detection is formulated as a binary classification task, and a classifier is trained for a target language model [37, 29, 26, 13]. For example, OpenAI trains its AI-text classifier with a RoBERTa-based model [29].
The developers collected samples from the WebText dataset<sup>2</sup> and labeled them as human-generated. Then, for each target GPT-2 model, they collected the generated samples and labeled them as machine-generated. Finally, they fine-tuned the pretrained RoBERTa-based model [29] for AI-text classification. More recently, with the appearance of CharGPT, OpenAI tuned a GPT model called AI-Classifier<sup>1</sup> using data from several sources. The human-written text comes from three sources: a new Wikipedia dataset, the WebText dataset collected in 2019, and a set of human demonstrations collected as part of training InstructGPT [24]. To collect machine-generated text, for the Wikipedia and WebText datasets, they truncated the articles sampled from the original corpus and used 34 models to generate article completion, pairing each generated text with the original article. For the demonstrations, they used a model to generate responses for each prompt and paired them with the corresponding human demonstrations. This detector was only accessible via a web interface since its release in January 2023, and it has been taken down since July 2023. (iii) Watermark methods: post-hoc watermarking techniques, such as rule-based methods [1, 15, 31] and deep-learning-based methods [6, 32], can be applied to an LLM. At inference time, [16] proposed a soft watermarking scheme to embed a watermark in each word of the generated sentence by dividing the vocabulary into different lists and sampling the next token in a differentiated manner. However, many existing AI-text detectors are shown to be significantly weakened by paraphrasing in [27].
Adversarial Learning for Natural Language Generation. The success of GAN [10] in the computer vision domain has motivated many studies in natural language generation. However, since text generation is a sequential sampling process that occurs in a discrete vocabulary space, it is difficult to directly train a text generator using back-propagation in an end-to-end manner [36, 7, 5, 35]. There are two common approaches to tackle this problem. The first one is to replace the discrete sampling operation with continuous approximation techniques [35, 5], such as Gumbel-Softmax [14, 22]. The second one is to view text generation as a decision-making process and cast the generator as a policy [36, 34, 7, 33]. A typical example is SeqGAN [36]. During generation, SeqGAN considers the generated tokens as the state and the next token to be generated as the action, and it adopts Monte Carlo search to collect reward signals from the discriminator. Instead of using a classifier as the discriminator, the Diversity-Promoting GAN [34] uses a unidirectional LSTM as the discriminator and combines both word-level and sentence-level rewards into training. TextGAIL [33] proposed an imitation learning paradigm in which the rewards of the human-written text are regarded as a constant value. Then, both the rewards from human-text and AI-text are used to optimize the generator with PPO. These works all used warm-up training for the generator with maximum likelihood estimation (MLE) on the probability of the generated text sequence. On the other hand, [7] trained a language GAN from scratch. Our proposed RADAR differs from these works in that we focus on training a robust AI-text detector with a tunable paraphraseer. Another line of work, such as [20, 4], uses paraphrasing techniques to find adversarial examples for natural language processing tasks and for training a robust language model via adversarial training. Their focus is on the correctness of natural language understanding, which is beyond our scope of AI-text detection.
3 RADAR: Methodology and Algorithms
We start this section by giving an overview and mathematical notations of our proposed RADAR framework in Figure 1. Then, in Sections 3.1 and 3.2, we provide the details on the design and training of the paraphraseer and detector, respectively. Finally, we will summarise the entire training process into an algorithmic procedure in Section 3.3.
High-Level Methodology. Our RADAR framework consists of three neural-network-based language models (LMs): the target LM  $\mathcal{T}_{\theta}$ , the detector  $\mathcal{D}_{\phi}$  and the paraphraseer  $\mathcal{G}_{\sigma}$ , parameterized with  $\theta$ ,  $\phi$  and  $\sigma$ , respectively. We note that  $\mathcal{T}_{\theta}$  is frozen (no updates on  $\theta$ ) in the entire process. We summarize RADAR into three key steps:
- Step 1 (Data preparation): Before training, we build  $\mathcal{M}$ , the corpus of AI-text, by applying document completion based on the prefix span of text in the human-text corpus  $\mathcal{H}$  using  $\mathcal{T}_{\theta}$ .
$^{2}$ https://huggingface.co/datasets/openwebtext
- Step 2 (Paraphraser update): We collect AI-text samples  $x_{m}$  from  $\mathcal{M}$  and use  $\mathcal{G}_{\phi}$  to do paraphrasing on  $x_{m}$  to generate paraphrased AI-text  $x_{p}$  to form a corpus  $\mathcal{P}$ . Then, we use the reward of  $x_{p}$  returned by the detector  $\mathcal{D}_{\theta}$  to update the paraphraseer  $\mathcal{G}_{\phi}$  using PPO.
- Step 3 (Detector update): We use the human-text samples  $x_{h}$  from  $\mathcal{H}$ , the original AI-text samples  $x_{m}$  from  $\mathcal{M}$ , and the paraphrased AI-text samples  $x_{p}$  from  $\mathcal{P}$  in step 2 to update the detector  $\mathcal{D}_{\theta}$  with a logistic loss function.
- Step 4 (Performance Validation and Evaluation): During training, we use the test set of WebText as the validation dataset to estimate RADAR's performance. For evaluation, we use  $\mathcal{T}_{\theta}$  to generate AI-text for the evaluation dataset and to calculate RADAR's detection AUROC.
Step 2 to Step 3 can be repeated until there is no improvement in the AUROC evaluated on the validation dataset. The nature of rivalry in adversarial learning and the introduced competition helps the detector to learn to be robust in detecting both original and paraphrased AI-text.
3.1 Training Paraphraser via Clipped PPO with Entropy Penalty
In RADAR, the goal of the paraphraseer  $\mathcal{G}_{\sigma}$  is to paraphrase the input machine-generated text  $x_{m}$ . We model the generation of paraphrased text as a decision-making process, taking  $x_{m}$  as the state and the output text  $x_{p}$  as the action. In particular, we optimize  $\mathcal{G}_{\sigma}$  using the reward feedback from the detector  $\mathcal{D}_{\phi}$  with PPO. The output of  $\mathcal{D}_{\phi}(x_{p})$  is the predicted likelihood of  $x_{p}$  being Human-text. The reward returned by  $x_{p}$  and the log probability of the text  $x_{p}$  are defined in Eq. 1:
$$
R \left(x _ {p}, \phi\right) = \mathcal {D} _ {\phi} \left(x _ {p}\right) \in [ 0, 1 ]; \log P _ {\mathcal {G} _ {\sigma}} \left(x _ {p} \mid x _ {m}\right) = \sum_ {i = 1} ^ {N} \log P _ {\mathcal {G} _ {\sigma}} \left(x _ {p} ^ {i} \mid x _ {m}, x _ {p} ^ {1: i - 1}\right), \tag {1}
$$
where  $x_{p}^{i}$  means the  $i$ -th token in the sentence  $x_{p}$  of length  $N$  and  $x_{p}^{1:i-1}$  represents the first  $i-1$  tokens in  $x_{p}$  ( $x_{p}^{1:0}$  means the default starting token).
We propose Clipped PPO with Entropy Penalty (cppo-ep) in RADAR to optimize  $\mathcal{G}_{\sigma}$ . Let  $\mathrm{clip}(\cdot, a, b)$  denote a value-clipping operation with a lower limit  $a$  and an upper limit  $b$ ,  $r(\sigma, x_m, x_p)$  be the importance sampling ratio between a new policy  $\mathcal{G}_{\sigma}$  and an old policy  $\mathcal{G}_{\sigma'}$ , and  $(x_m, x_p) \sim P_{\mathcal{G}_{\sigma'}}$  be a state-action pair sampled from  $\mathcal{G}_{\sigma'}$ . The loss of cppo-ep is defined as:
$$
L _ {\mathcal {G}} (\sigma) = \mathbb {E} _ {(x _ {m}, x _ {p}) \sim P _ {\mathcal {G} _ {\sigma^ {\prime}}}} \underbrace {- \min  \left\{\operatorname {c l i p} (r (\sigma , x _ {m} , x _ {p}) , 1 - \epsilon , 1 + \epsilon) , r (\sigma , x _ {m} , x _ {p}) \right\} \cdot A (x _ {p} , \phi)} _ {L _ {\mathcal {A}}} \underbrace {- \gamma S (\sigma)} _ {L _ {\mathcal {E}}} \tag {2}
$$
where  $\mathbb{E}$  denotes expectation,  $\epsilon$  is a parameter used in clipping to avoid the importance ratio  $r$  from being too large,  $A(x_{p},\phi)$  is the advantage item of the paraphrased text  $x_{p}$  obtained by applying normalization to  $R(x_{p},\phi)$  across the entire PPO sample buffer  $\mathcal{B}$ .  $S(\sigma) = \mathbb{E}_{(x_m,x_p)\sim P_{\mathcal{G}_{\sigma '}}} - P_{\mathcal{G}_{\sigma}}(x_p|x_m)\log P_{\mathcal{G}_{\sigma}}(x_p|x_m)$ , which is an entropy term introduced to encourage  $\mathcal{G}_{\sigma}$  to explore more diverse generation policy.  $\gamma$  is a coefficient to control the ratio between  $L_{\mathcal{A}}$  and  $L_{\mathcal{E}}$ , in order to make a balance between advantage  $(L_{\mathcal{A}})$  and diversity  $(L_{\mathcal{E}})$  when paraphrasing.
3.2 Training Detector via Reweighted Logistic Loss
In a typical GAN training process, the discriminator receives an equal amount of positive and negative samples in each step, assuring an in-batch sample balance. However, in RADAR, by construction, the number of AI-text samples is twice the number of human-text samples, because each  $x_{h}$  from the human-text corpus  $\mathcal{H}$  is paired with a sample  $x_{m}$  from the original AI-text corpus  $\mathcal{M}$  as well as a paraphrased sample  $x_{p}$  generated by the paraphrase  $\mathcal{G}_{\phi}$ . To handle this in-batch imbalance problem, we use a reweighted logistic loss function to optimize the detector  $D_{\phi}$ , as described in Eq. 3:
$$
L _ {\mathcal {D}} (\phi) = \underbrace {- \mathbb {E} _ {x _ {h} \sim \mathcal {H}} \log \mathcal {D} _ {\phi} (x _ {h})} _ {L _ {\mathcal {H}}: \text {l o s s o n h u m a n - t e x t}} + \lambda \underbrace {\mathbb {E} _ {x _ {m} \sim \mathcal {M}} - \log \left(1 - \mathcal {D} _ {\phi} (x _ {m})\right)} _ {L _ {\mathcal {M}} ^ {1}: \text {l o s s o n o r i g i n a l A I - t e x t}} + \lambda \underbrace {\mathbb {E} _ {x _ {m} \sim \mathcal {M}} - \log \left(1 - \mathcal {D} _ {\phi} (\mathcal {G} _ {\sigma} (x _ {m}))\right)} _ {L _ {\mathcal {M}} ^ {2}: \text {l o s s o n p a r a p h r a s e d A I - t e x t}} \tag {3}
$$
Recall that  $\mathcal{D}_{\phi}(x) \in [0,1]$  is the predicted probability of an input instance  $x$  being Human-text.  $L_{\mathcal{H}}$  is the loss to improve the correctness of predicting  $x_h \sim \mathcal{H}$  as human-written.  $L_{\mathcal{M}} = L_{\mathcal{M}}^1 + L_{\mathcal{M}}^2$ , where  $L_{\mathcal{M}}^1$  and  $L_{\mathcal{M}}^2$  are used to avoid  $x_m$  and  $x_p$  from being predicted as human-text, respectively.  $\lambda$  is a coefficient ranging from 0 to 1. We introduce  $\lambda$  to adjust the proportion of AI-text components in the overall loss function to alleviate the effects of sample imbalance.
3.3 RADAR Algorithm
The entire training procedure of RADAR is summarized in Algorithm 1. For a given target LLM, RADAR returns a trained paraphraseer and a trained detector through the designed training steps. In the evaluation phase, the detector is used to predict the likelihood of AI-text for any input instance.
Algorithm 1 RADAR: Robust AI-Text Detection via Adversarial Learning
1: Data initialization:  
2: Collect human-written text to build human-text corpus  $\mathcal{H}$   
3: Select a target language model  $\mathcal{T}_{\theta}$  to perform document completion on  $\mathcal{H}$  to build the corresponding AI-text corpus  $\mathcal{M}$   
4: Build a replay buffer  $\mathcal{B}$  to store samples temporarily collected for training  
5: Build a validation dataset  $\mathcal{V}$  from  $\mathcal{H}$  and  $\mathcal{M}$   
6: Model initialization:  
7: Detector  $\mathcal{D}_{\phi} \gets \phi_{\text{pretrain}}$  (a pretrained language model)  
8: Paraphrase  $\mathcal{G}_{\sigma} \gets \sigma_{\text{pretrain}}$  (a pretrained language model)  
9: Model training:  
10: for  $i = 1$ : maximum step do  
11: Sample  $x_{h}$  and its corresponding  $x_{m}$  from  $\mathcal{H}$  and  $\mathcal{M}$  respectively  
12: Use  $\mathcal{G}_{\sigma}$  to paraphrase  $x_{m}$  and generate  $x_{p}$   
13: Collect reward  $R(x_{p}, \phi)$  as in Eq. 1  
14:Normalize  $R(x_{p}, \phi)$  to compute the advantage function  $A(x_{p}, \phi)$  used in Eq. 2  
15: Fill  $\mathcal{B}$  with  $(x_{h}, x_{m}, x_{p}, A(x_{p}, \phi))$   
16:  $\sigma' \gets \sigma$  # initialize the old policy  $\sigma'$  as the current policy  $\sigma$   
17: for  $(x_{h}, x_{m}, x_{p}, A(x_{p}, \phi)) \in \mathcal{B}$  do  
18: Compute the log probability  $\log P_{\mathcal{G}_{\sigma}}(x_{p} | x_{m})$  and  $\log P_{\mathcal{G}_{\sigma}'}(x_{p} | x_{m})$  using Eq. 1  
19: Update  $\mathcal{G}_{\sigma}$  using Eq. 2  
20: end for  
21: for  $(x_{h}, x_{m}, x_{p}, A(x_{p}, \phi)) \in \mathcal{B}$  do  
22: Update  $\mathcal{D}_{\phi}$  using Eq. 3  
23: end for  
24: Clear  $\mathcal{B}$   
25: Evaluate AUROC of  $\mathcal{D}_{\phi}$  on the validation dataset  $\mathcal{V}$   
26: end for  
27: Detector  $\mathcal{D}_{\phi} \gets \phi_{\text{best}}$  (the detector model with the best AUROC on the validation dataset)  
28: Paraphrase  $\mathcal{G}_{\sigma} \gets \sigma_{\text{best}}$  (the paraphrase model which pairs with  $\phi_{\text{best}}$ )  
29: Return  $\mathcal{D}_{\phi}$  and  $\mathcal{G}_{\sigma}$
4.1 Experimen Setup
Datasets and Metrics. For training, we sampled 160K documents from WebText [9] to build the human-text corpus  $\mathcal{H}$ . Then, we build the original AI-text corpus  $\mathcal{M}$  from  $\mathcal{H}$  using a target language model  $\mathcal{T}_{\theta}$ , which performs text completion using the first 30 tokens as the prompt and limits the sentence length to be 200 tokens. For evaluation, we select four human-text datasets covering different domains. Following [23], we use Xsum, SQuAD, and Reddit WritingPrompts (WP) to test a detector's ability to detect fake news, avoid academic fraud, and identify machine-generated literature innovation, respectively. In addition, we also use the non-native-authored TOEFL dataset (TOFEL) [21] to evaluate a detector's bias when encountering non-native-authored English text. Please see Appendix A for more details about the evaluation datasets. Following existing works, we report the area under the receiver operating characteristic curve (AUROC) score by varying the detector's threshold as the performance measure (higher is better), which captures the relationship between the true positive rate and the false positive rate.