SlowGuess/ACL_2020_OCR huggingface dataset:
Daphne Ippolito†‡*
daphnei@seas.upenn.edu
Daniel Duckworth*
duckworthd@google.com
Chris Callison-Burch†‡
ccb@seas.upenn.edu
Douglas Eck‡
deck@google.com
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions. Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies—top-k, nucleus sampling, and untruncated random sampling—and show that improvements in decoding methods have primarily optimized for fooling humans. This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems. We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over  $30\%$  of the time. Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.
State-of-the-art generative language models are now capable of producing multi-paragraph excerpts that at a surface level are virtually indistinguishable from human-written content (Zellers et al., 2019; Radford et al., 2019; Adelani et al., 2020). Often, only subtle logical fallacies or idiosyncrasies of language give away the text as machine-generated, errors that require a close reading and/or domain knowledge for humans to detect.
Deceptive text, whether human- or machine-generated, has entered the sphere of public concern (Cooke, 2018). It propogates quickly (Vosoughi et al., 2018), sets political agendas
(Vargo et al., 2018), influences elections (Allcott and Gentzkow, 2017), and undermines user trust (Wang et al., 2012; Song et al., 2015). Recently, Adelani et al. (2020) have shown that automatically generated reviews are perceived to be as fluent as human-written ones. As generative technology matures, authors, well-meaning or otherwise, will increasingly employ it to augment and accelerate their own writing. It is more imperative now than ever for both humans and automated systems to be able to detect and identify machine-generated texts in the wild. However, there has thus been little inquiry into the textual properties that cause humans to give generated text high human-like ratings compared to those that cause automatic systems to rate it highly.
To speak of texts produced by language models, we must first consider how these texts are generated. A neural language model encodes a probability distribution over the next word in a sequence given the previous words.<sup>1</sup> A decoding strategy is an algorithm that generates sequences from a language model by determining how words should get selected from this distribution. The field has largely moved toward probabilistic decoding strategies that randomly sample from the output distribution token-by-token. However, when many low-likelihood words cumulatively contain quite a bit of probability mass, choosing one of these words can lead to odd or contradictory phrases and semantic errors. Humans are quick to notice these types of errors.
For this reason, it has become common to modify the language model's output probability distribution to increase the chance of sampling tokens with high likelihood according to the language model. Top- $k$  random sampling, where low-likelihood words are restricted from being
*Equal contribution, ‡Google, †University of Pennsylvania
Often these 'words' are actually subword character sequences such as BPE tokens (Sennrich et al., 2016).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808-1822 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
generated, is one such method. A language model that is only permitted to produce high-likelihood words is less likely to make a poor choice and create the type of mistakes that are easy for humans to detect. Since humans are not proficient at identifying when a model subtly favors some utterances more often than a human author would, they don't notice the over-representation of high-likelihood words in the generated text. In contrast, automatic systems excel at identifying statistical anomalies and struggle to build deeper semantic understanding. Top- $k$  in particular creates text that is easy for machines to detect but very hard for humans. Thus, we observe the general trend: as the number of unlikely words available to be chosen is increased, humans get better at detecting fakes while automatic systems get worse.
In this work, we study three popular random decoding strategies—top- $k$ , nucleus, and temperature sampling—applied to GPT-2 (Radford et al., 2019). We draw a large number of excerpts generated by each strategy and train a family of BERT-based (Devlin et al., 2019) binary classifiers to label text excerpts as human-written or machine-generated. We find large differences in human rater and classifier accuracy depending on the decoding strategy employed and length of the generated sequences. Regardless of strategy, we find human raters achieve significantly lower accuracy than the automatic discriminators. We also show that when a decoding strategy severely modifies the unigram token distribution, as top- $k$  does, humans have trouble detecting the resultant generated text, but automatic classifiers find it the easiest to discriminate. Worryingly, we further find that classifiers are brittle; they generalize poorly when trained to discriminate samples from one strategy and then evaluated on samples from another.
- A comprehensive study of generated text detection systems' sensitivity to model structure, decoding strategy, and excerpt length.
- An analysis of human raters' ability to identify machine-generated content, and how human raters differ from automatic detectors.
Generative Language Models With a sufficiently large training set and number of trainable parameters, neural language models based on the
Transformer architecture (Vaswani et al., 2017) are capable of generating convincing, human-like excerpts up to several paragraphs in length. GPT-2 (Radford et al., 2019), GROVER (Zellers et al., 2019), and Transformer-DMCA (Liu et al., 2018) are a few examples of large, publicly available models with this ability. GROVER, in particular, has been shown to generate fake news that is more trustworthy than human-written fake news according to human raters.
Human Detection The task of trying to guess whether text is coming from a robot or a fellow human was made famous by the Turing Test (Turing, 1950). It continues to be used in chatbot evaluation (Lowe et al., 2017). The related (but not identical) task of asking human raters to judge the quality of machine-generated excerpts remains the gold-standard for evaluating open-domain generation systems (van der Lee et al., 2019). Kreps et al. (2020), Gehrmann et al. (2019), and others have stressed the importance of humans being able to identify fake content on the web.
Automatic Detection The rise of machine-generated content has led to the development of automated systems to identify it. GROVER was designed to not only generate convincing news excerpts but to also identify them using a fine-tuned version of the generative model itself (Zellers et al., 2019). GLTR, expecting attackers to use sampling methods that favor high-likelihood tokens, aims to make machine-generated text detectable by computing histograms over per-token log likelihoods (Gehrmann et al., 2019). Bakhtin et al. (2019) frame human-text detection as a ranking task and evaluate their models' cross-domain and cross-model generalization, finding significant loss in quality when training on one domain and evaluating on another. Schuster et al. (2019) argue that the language distributional features implicitly or explicitly employed by these detectors are insufficient; instead, one should look to explicit fact-verification models. Finally, discriminators for whether text is machine-generated are a promising research direction in adversarial training (Lin et al., 2017; Li et al., 2017) and in automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).
Natural Language Understanding Automatic detection of machine-generated text benefits from a semantic understanding of the text. Contradic-
tions, falsehoods, and topic drift can all indicate that an excerpt was machine-generated. Encoder-only Transformer models such as BERT (Devlin et al., 2019) have been shown to do very well at tasks requiring this understanding. While we fine-tune BERT for the task of classifying whether text was machine-generated, others have used the contextual word embeddings from a pre-trained BERT model without fine-tuning to compute a quality score for generated text (Zhang et al., 2020). It is worth noting that recent work has raised questions as to whether BERT truly builds a semantic understanding to make its predictions, or whether it merely takes advantage of spurious statistical differences between the text of different classes (Niven and Kao, 2019).
We frame the detection problem as a binary classification task: given an excerpt of text, label it as either human-written or machine-generated. In particular, we are interested in how variables such as excerpt length and decoding strategy impact performance on this classification task. We thus create several datasets. Each is approximately balanced between positive examples of machine-generated text and negative examples of human-written text. While they all share the same human-written examples, each dataset contains a different set of machine-generated examples sampled using one particular decoding strategy. We also build additional datasets by truncating all of the examples to a particular sequence length,
By training a separate classifier on each dataset, we are able to answer questions about which decoding strategy results in text that is the easiest to automatically disambiguate from human-written text. We are also able to answer questions about how the length of the examples in the training set impacts our ability to automatically classify excerpts of that same length as either human-written or machine-generated.
4 Dataset Methodology
All of our generated text samples are drawn from GPT-2, a state-of-the-art Transformer-based generative language model that was trained on text from popular web pages (Radford et al., 2019). While we use the GPT-2 LARGE model with 774M parameters, we found that similar trends to those reported here hold in experiments with
smaller language models.
Given an autoregressive language model that defines a probability distribution over the next token given the previous tokens in a sequence, a decoding strategy generates text by deciding how to output a token at each step based on the predicted distributions. Perhaps the most straightforward decoding strategy is to randomly choose a token with probability proportional to its likelihood. A challenge with the random sampling approach is that these probability distributions often contain a long tail of vocabulary items that are individually low-probability but cumulatively comprise a substantial amount of probability mass. Holtzman et al. (2020) observe that choosing tokens from this tail often leads to incoherent generations.
Top- $k$  sampling, nucleus sampling, and (in the extreme) beam search have all been proposed to heuristically promote samples with higher pertoken likelihoods. Top- $k$  and nucleus sampling both do so by setting the likelihood of tokens in the tail of the distribution to zero. Top- $k$  restricts the distribution to all but the  $k$  most likely tokens, where  $k$  is a constant (Fan et al., 2018). Nucleus sampling, also called top- $p$ , truncates the distribution at each decoding step  $t$  to the  $k_{t}$ -most-likely next tokens such that the cumulative likelihood of these tokens is no greater than a constant  $p$  (Holtzman et al., 2020).
We thus consider three different decoding strategy settings:
- Sample from the untruncated distribution
- Top- $k$ , choosing  $k = 40$  (Radford et al., 2019).
- Nucleus sampling (aka top- $p$ ), choosing  $p = 0.96$  (Zellers et al., 2019).
In addition, we form "negative" examples of human-written text by taking excerpts of web text that come from the same distribution as GPT-2's training data. By picking text that resembles GPT-2's train set, we ensure that our classifiers can't simply take advantage of stylistic differences between the human-written text corpus and the kind of text GPT-2 was trained to generate.
For each decoding method, we construct a training dataset by pairing 250,000 generated samples with 250,000 excerpts of web text. 5,000 additional paired samples are kept aside for validation and test datasets. Lastly, we filter out excerpts with fewer than 192 WordPiece tokens (Wu et al.,
$^{2}$ https://github.com/openai/gpt-2-output-dataset
2016) (excerpts might be quite short if the model produces an end-of-text token early on). See Appendix 1 for final dataset sizes.
A crucial question when generating text with a language model is whether or not to provide a priming sequence which the language model should continue. Unconditioned samples, where no priming text is provided, in conjunction with top- $k$  sampling, lead to pathological behavior for discriminators as the first token of the generated text will always be one of  $k$  possible options. On the other hand, if long sequences of human text are used as priming, the space of possible generated sequences is larger, but the detection problem shifts from one of "how human-like is the generated text?" to "how well does the generated text follow the priming sequence?".
Since in this study we are interested in the former simpler question, we create two datasets, one with no priming, and one with the minimum amount of priming possible: a single token of web text. This means that for every excerpt of web text in the training set, there is an excerpt of machine-generated text that starts with the same token. We find that even with limited priming, the ability of automatic detectors can be strongly impacted.
To study the effect of excerpt length, we construct variations of the above datasets by truncating all excerpts to ten possible lengths ranging from 2 to 192 WordPiece tokens (Wu et al., 2016). In total, we obtain sixty dataset variations: one per sampling method, truncation length, and choice of priming or no priming.
5 Automatic Detection Method
The primary discriminator we employ is a fine-tuned BERT classifier (Devlin et al., 2019). We fine-tune one instance of BERT per dataset variation described above. For the longest sequence length,  $n = 192$ , we compare BERT's performance with several simple baselines that have been proposed in other work.
Fine-tuned BERT We fine-tune BERT-LARGE (cased) on the task of labeling a sentence as human- or machine- generated. The models are trained for 15 epochs, with checkpoints saved every 1000 steps, and a batch size of 256. All results are reported on the test set using the checkpoint for which validation accuracy was highest.
Bag-of-Words For each sequence, we compute a bag-of-words embedding where each dimension
corresponds to a token in GPT-2's 50,000 token BPE vocabulary (Sennrich et al., 2016), and we count how many times that token appears in the text sequence. We then train a logistic regression binary classifier to predict human- or machine-written given this 50,000-dimensional embedding. We experimented with truncating embedding size by removing entries for infrequent vocabulary words, but this did not improve performance.
Histogram-of-Likelihood Ranks Following GLTR (Gehrmann et al., 2019), we compute the probability distribution of the next word given the previous words in a text sequence according to a trained language model (in our case the same GPT-2 model that was used for generation). At each sequence position, we rerank the vocabulary words by likelihood, and record the rank of the ground-truth next word within this list. These ranks are then binned. GLTR uses four bins, counting (1) the number of times the top 1 word is seen, (2) the number of times words ranked 2 through 5 are seen, (3) words ranked 6-100, and (4) words ranked  $>100$ . However, we observe higher accuracy when 50 bins are spread uniformly over the possible rankings. This means that since there are 50,000 vocabulary words, the first bin counts the number of times the actual next word was within the 1,000 mostly likely next words, the second bin counts the 1,001-2,000th, and so on. We then train logistic regression binary classifiers to predict human- or machine-written given either the 4-dimensional histograms or 50-dimensional histograms as input.
Total Probability Solaiman et al. (2019) propose a very simple baseline consisting of a threshold on the total probability of the text sequence. An excerpt is predicted as machine-generated if its likelihood according to GPT-2 is closer to the mean likelihood over all machine-generated sequences than to the mean of human-written ones.
6 Human Detection Method
The human evaluation task is framed similarly to the automatic one. We ask the raters to decide whether a passage of text was written by a human or by a computer algorithm. (Full instructions are in the Appendix.) Raters are allowed to choose between four options: "definitely" or "possibly" machine-generated and "definitely" or "possibly" human-written. They are first shown an excerpt of length 16 WordPiece tokens. After they make
Method	BERT	BagOfWords	HistGLTRBuckets	Hist50Buckets	TotalProb acc	Human acc
acc	AUC	acc	AUC	acc	AUC	acc	AUC
k40-1wordcond	0.88	0.99	0.79	0.87	0.52	0.52	0.69	0.76	0.61	0.64
p0.96-1wordcond	0.81	0.89	0.60	0.65	0.53	0.56	0.54	0.56	0.63	0.77
p1.0-1wordcond	0.79	0.92	0.59	0.62	0.53	0.55	0.54	0.55	0.65	0.71
Table 1: Performance (accuracy and AUC) of the fine-tuned BERT classifier and several simple baselines on detecting length-192 sequences generated with one word of priming (1worccond). Note that p1.0 refers to untruncated random sampling, where we sample from  $100\%$  of the probability mass. The last column shows human performance on the same task where accuracy with a  $50\%$  baseline is computed by randomly pairing samples from each decoding strategy with a human-written sample.
a guess, the length of the excerpt is doubled, and they are asked the same question again. This continues until the entire passage of length 192 tokens is shown. Passages are equally likely to be human-written or machine-generated, with the machine-generated excerpts being evenly split between the three sampling strategies considered in this paper.
Initially, Amazon Mechanical Turk (AMT) raters were employed for this task, but rater accuracy was poor with over  $70\%$  of the "definitely" votes cast for "human" despite the classes being balanced. Accuracy, even for the longest sequences, hovered around  $50\%$ . The same study was then performed with university students who were first walked through ten examples (see Appendix Table 4) as a group. Afterward, they were asked to complete the same tasks that had been sent to the AMT workers. No additional guidance or direction was given to them after the initial walk-through. We will refer to this group as the "expert raters." Among them,  $52.1\%$  of "definitely" votes were cast for human, and accuracy on the longest excerpt length was over  $70\%$ .
The human evaluation dataset consisted of 150 excerpts of web text and 50 excerpts each from the three decoding strategies. Each question was shown to at most three raters, leading to 900 total annotations from the untrained workers and 475 from the expert raters. A more detailed breakdown can be found in the Appendix.
7 Automatic Detection Results
Simple Baselines Table 1 shows the performance of the baseline discriminators on length-192 sequences, as compared with fine-tuned BERT. Reassuringly, BERT far surpasses all simple baselines, indicating that it is not fully possible to solve the detection problem without complex sequence-based understanding. The simplest baseline, TotalProb, which makes a decision based on the likelihood of the sequence, performs sur
prisingly well (over  $60\%$  accuracy for all sampling methods) relative to the methods which involve training logistic regression models.
Logistic regression on bag-of-words is the best of the baselines, beating out the histogram-based methods. While Gehrmann et al. (2019) report an AUC of 0.87 on classifying text as real or generated using logistic regression on the four buckets of the GLTR system, we report AUC between 0.52 and 0.56 for this task. The discrepancy is likely due to the fact that the human-written text in our discriminator training set comes from the same distribution as the text used to train the language model, while in GLTR the human text comes from children's books, scientific abstracts, and newspaper articles. The selection of training data for learned detection systems is crucial. In real-world applications, the choice ought to reflect the genres that builders of text-generation systems are trying to impersonate.
Fine-tuned BERT In Figure 1a, we begin by observing discriminator accuracy as a function of excerpt length and sampling method. As can be intuitively expected, as sequence length increases, so too does accuracy. For unconditioned text decoded with nucleus (p0.96) and untruncated (p1.0) random sampling, we find discriminator accuracy increases from  $55\%$ , near random, to about  $81\%$  for the longest sequences tested. In contrast, discriminators trained and evaluated on top- $k$  achieve over  $80\%$  accuracy even on 16-token excerpts.
Why are top- $k$ 's samples so easy to detect? In Figure 2b, we see the percentage of probability mass concentrated in the  $k$  most common token types for each sampling method. While random sampling and nucleus sampling are very similar to human-written texts, we see top- $k$  concentrating up to  $80\%$  of its mass in the first 500 most common tokens. The other sampling methods as well as human-written texts require at least 1,100 token types for the same. It is clear that top- $k$ 's distribu
(b)
Figure 1: In (a), accuracy increases as the length of the sequences used to train the discriminator is increased. In (b), we see that the BERT fine-tuned discriminator predicts about the same number of false-positives as false-negatives when trained with samples generated using top- $p$  sampling. However, for top- $k$ , it more often mistakes machine-generated text to be human-written, while for untruncated random sampling the opposite is the case.
tion over unigrams strongly diverges from human-written texts—an easy feature for discriminators to exploit. In fact, See et al. (2019) note that it takes setting  $k$  to 1000 to achieve about the same amount of rare word usage and fraction of non-stopword text as as human writing. $^3$  This makes it very easy for the model to pick out machine-generated text based on these distributional differences.
One way to help resolve this problem is to add priming text. Doing so causes more rare words to be incorporated into the top-  $k$  of the unigram distribution. Adding even a single human word of priming significantly reduces the performance of detectors trained with top-  $k$  random sampling. Without priming, a discriminator trained on sequences of length 2 can classify with  $\sim 90\%$  accuracy the provenance of the text (Figure 1a). By adding one priming token, accuracy drops to  $\sim 65\%$ . Even on the longest 192-length sequences, top-  $k$  discriminator accuracy is  $6\%$  lower on the primed dataset than the unprimed one.
When generating with nucleus or untruncated random sampling, adding a priming token is not as impactful, as these methods are already sampling from a large fraction (or all) of the probability distribution. This is seen in Figure 2a where at the very first step of unprimed generation, nucleus sampling selects from 3075 possible vocabulary words, and at later positions selects from on