SlowGuess/ACL_2020_OCR huggingface dataset:
average more than 500. Untruncated random sampling always selects from the entire 50,000 word vocabulary, whereas top- $k$  only selects from  $k$ .
Transferability In Table 2, we show how discriminators trained with samples from one decoding strategy can transfer at test time to detecting samples generated using a different decoding strategy. Unsurprisingly a discriminator trained on top- $k$  generalizes poorly to other sampling methods: accuracy drops to as low as  $42.5\%$ , worse than chance. Conversely, training the discriminator with sequences sampled from the untruncated distribution leads to little transferability to detecting top- $k$  samples. Only the discriminator trained with nucleus sampling (a compromise between unmodified sampling and top- $k$ ) was able to detect sequences from the other sampling strategies without too much of a hit to accuracy. As expected, a discriminator trained on an equal portion of data from each decoding method does reasonably at detecting all three.
Perhaps this lack of transferability is related to each discriminator's calibration. Indeed, the degree to which a discriminator's average prediction deviates from  $50\%$  is a direct indicator of its accuracy. In Table 3, we observe that of the three BERT discriminators, only that trained on top- $p$  samples predicts 'machine-generated' on approximately  $50\%$  of in-domain examples as expected. This same discriminator's behavior holds on datasets generated by other sampling strategies
3when decoding from the GPT-2 small model with 117M parameters.
Figure 2: In (a), the average (over sequences in the test set)  $k$  chosen at each step during generating with nucleus sampling is plotted. Adding a single word of priming strongly impacts the  $k$ s chosen for the first few positions, but this difference quickly dissipates. In (b), we consider the first token generated in each sequence by top-  $k$ , and plot what fraction of these are captured by the  $k$  most common unique tokens from the vocabulary. Overall, at its first step, top-  $k$  concentrates  $80\%$  of its probability mass in the 500 most common tokens from the vocabulary.
(a)
(b)
Figure 3: (a) and (b) show human rater accuracy of correctly identifying an excerpt as human-written or machine-written, shown with  $80\%$  confidence internals, in (a), broken up by decoding strategy and in (b), overall. Accuracy increases as raters observe more tokens. (c) shows that for short excerpts, most rater mistakes are them incorrectly thinking machine-generated text is human written. The two errors types become more balanced at longer lengths.
Eval
top-k	nucleus	random
Train	top-k	90.1	57.1	43.8
nucleus	79.1	81.3	78.4
random	47.8	63.7	81.7
mixed	88.7	74.2	72.2
as well. In contrast, we observe that discriminators trained on top-k and untruncated random samples severely underestimate the percentage of machine-generated excerpts in out-of-domain datasets. Even within domain (Figure 1b), we find both discriminators heavily favor a single class, in
Table 2: Accuracy of BERT fine-tuned discriminator when trained on samples from one strategy (rows) and evaluated on another (columns). Trained on samples with 192 tokens. The 'mixed' dataset is one containing an equal portion of samples from each strategy.
Eval
top-k	nucleus	random
Train	top-k	60.9	27.9	14.5
nucleus	49.2	51.7	48.9
random	7.3	22.6	38.3
Table 3: Average probability of 'machine-generated' according to each length-192 discriminator. The expected in-domain probability is 0.5. One token of conditioning.
creasingly so as the number of tokens increases.
Human Evaluation Overall human performance across all sampling methods is shown in Figure 3b. Even with the multi-paragraph 192-length excerpts, human performance is only at  $71.4\%$ , indicating that even trained humans struggle to correctly identify machine-generated text over a quar
Table 4: Some 192-token examples where at least two expert raters agreed with each other, but were not in agreement with the automatic discriminators. The first row shows examples where the ground-truth was human-written, the second shows machine-generated examples where the corresponding discriminator guessed incorrectly, and the third shows machine-generated examples where the discriminator was correct, but raters got it wrong.
ter a time. However, it is worth noting that our best raters achieved accuracy of  $85\%$  or higher, suggesting that it is possible for humans to do very well at this task. Further investigation is needed into how educational background, comfort with English, participation in more extensive training, and other factors can impact rater performance.
To break up the accuracies by sampling method in a way that is comparable to the results shown for the automatic discriminators, we pair each machine-generated example with a randomly selected one of webtext to create a balanced dataset for each sampling strategy. Performance is shown in Figure 3a. Top-  $k$  produces the text that is hardest for raters to correctly distinguish, but as shown in Section 7, it is the easiest for our automatic detection systems. Samples from untruncated random sampling and nucleus sampling with  $p = 0.96$  are equivalently difficult for raters to classify as machine-generated. Our human evaluation results suggest that much lower  $p$ -values than the 0.92 to 0.98 range proposed in Zellers et al. (2019) might be necessary in order to generate text that is considered significantly more human-like to human raters than the text produced by using the untrun
cated distribution.
Table 4 gives several examples where human raters and our BERT-based discriminators disagreed. When raters incorrectly labeled human-written text as machine-generated, often the excerpts contained formatting failures introduced when the HTML was stripped out. In the middle two examples, topic drift and falsehoods such as Atlanta being the "information hub of the nation's capital" allowed humans to correctly detect the generated content. However, in the bottom two examples, the high level of fluency left human raters fooled.
Overall we find that human raters—even "expert" trained ones—have consistently worse accuracy than automatic discriminators for all decoding methods and excerpt lengths. In our experiments, randomly-selected pairs of raters agree with each other on a mere  $59\%$  of excerpts on average. (In comparison, raters and discriminators agree on  $61\%$  to  $70\%$  of excerpts depending on the discriminator considered). We surmise that the gap between human and machine performance will only grow as researchers inevitably train bigger, better detection models on larger amounts of
training data. While improved detection models are inevitable, it is unclear how to go about improving human performance. GLTR proposes providing visual aids to humans to improve their performance at detecting generated-text, but it is unlikely that their histogram-based color-coding will continue to be effective as generative methods get better at producing high-quality text that lacks statistical anomalies.
In this work, we study the behavior of automated discriminators and their ability to identify machine-generated and human-written texts. We train these discriminators on balanced binary classification datasets where all machine-generated excerpts are drawn from the same generative model but with different decoding strategies. We find that, in general, discriminators transfer poorly between decoding strategies, but that training on a mix of data from methods can help. We also show the rate at which discriminator accuracy increases as excerpts are lengthened.
We further study the ability of expert human raters to perform the same task. We find that rater accuracy varies wildly, but has a median of  $74\%$ , which is less than the accuracy of our best-performing discriminator. Most interestingly, we find that human raters and discriminators make decisions based on different qualities, with humans more easily noticing semantic errors and discriminators picking up on statistical artifacts. In our experiments, these artifacts are most prominent with top- $k$  sampling. However, any strategy that oversamples high-likelihood words is susceptible. As the  $p$  in nucleus sampling is set increasingly lower to achieve more fluent text (some systems are already using  $p$  as low as 0.5 (Miculicich et al., 2019)), the distributional deviations that plague top- $k$  text will surface in nucleus sampling as well.
Holtzman et al. (2020) explain how a unique attribute of human language is that it dips in and out of low probability zones. This variance in likelihood is what makes human-written text interesting and exciting to read. Today's generation systems have not yet solved the problem of mimicking the human cadence without introducing poor word choices that are easy for humans to detect. Generation systems often optimize for fooling humans without acknowledging the trade-off that exists between human perception of quality and ease
of automatic detection. We therefore suggest three prongs for future research:
1. Identifying ways to improve the language models and decoding strategies we use in order to generate text that is both exciting (ie. unlikely) and semantically plausible.
2. Building better world understanding into automatic discriminators so that they are more capable of detecting the types of errors that humans notice.
3. Developing tools and educational materials to improve humans' ability to detect machine-generated text. These may include automatic detectors with components that explain their predictions.
Finally, we would like to note that all of our experiments were performed with English language models, and it remains an open question how the trade-off between ease of human detection and ease of automatic detection might differ for languages that are very different from English.
This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
We also thank Noah Fiedel, Peter Liu, Sharan Narang, Joao Sedoc, Yun William Yu, and Hugh Zhang for their valuable feedback.
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In International Conference on Advanced Information Networking and Applications, pages 1341-1354. Springer.
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives, 31(2):211-36.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc'Aurelio Ranzato, and Arthur Szlam. 2019.
Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351.
Nicole A Cooke. 2018. Fake news and alternative facts: Information literacy in a post-truth era. American Library Association.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111-116.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. arXiv preprint arXiv:1701.08198.
Sarah E Kreps, Miles McCain, and Miles Brundage. 2020. All the news that's fit to fabricate: A-generated text as a tool of media misinformation. Social Science Research Network.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355-368.
Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, pages 3155-3165.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic tuning test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1116-1126.
Lesly Miculicich, Marc Marone, and Hany Hassan. 2019. Selecting, planning, and rewriting: A modular approach for data-to-document generation and translation. EMNLP-IJCNLP 2019, page 289.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Tal Schuster, Roei Schuster, Darsh J Shah, and Regina Barzilay. 2019. Are we safe yet? the limitations of distributional features for fake news detection. arXiv preprint arXiv:1908.09805.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843-861.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
Jonghyuk Song, Sangho Lee, and Jong Kim. 2015. Crowdtarget: Target-based detection of crowdturfing in online social networks. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 793-804. ACM.
Alan Turing. 1950. Computing machinery and intelligence-am Turing. Mind, 59(236):433.
Chris J Vargo, Lei Guo, and Michelle A Amazeen. 2018. The agenda-setting power of fake news: A big data analysis of the online media landscape from 2014 to 2016. New media & society, 20(5):2028-2049.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146-1151.
Gang Wang, Christo Wilson, Xiaohan Zhao, Yibo Zhu, Manish Mohanlal, Haitao Zheng, and Ben Y Zhao. 2012. Serf and turf: crowdturfing for fun and profit. In Proceedings of the 21st international conference on World Wide Web, pages 679-688. ACM.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616.
Tianyi Zhang, Varsha Kishore, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
A.1 Dataset Sizes
Table 5 shows the number of sequences used for training and evaluating each of the automatic discriminators. Recall that each discriminator is trained for binary classification on an a dataset of machine-generated (positive) and human-written (negative) examples. Each dataset was constructed by pairing the human-written excerpts (last row of Table 5) with the machine-generated excerpts drawn via a particular decoding algorithm ('k40', 'p0.96', or 'p1.0') and priming strategy ('nocond' or '1wordcond'). Originally the human-written set and each machine-generated set contained 250,000 training examples, 5,000 validation examples, and 5,000 test examples. Table 5 shows the resulting counts after after all excerpts with sequence length shorter than 192 tokens were filtered out. Thus, the final training, validation, and test sets were almost, but not quite, balanced.
A.2 Further Details on Human Evaluation
The user interface for the human evaluation task is shown in Figure 6. At each step, the rater is shown additional text and asked to guess whether the excerpt is human-written or machine-generated. They are able to revise their guess at each subsequent step. The newly appended text at each step is bolded in the UI. At the end, workers are told whether or not they got the question correct.
To gauge worker attention levels,  $10\%$  of questions shown to workers explicitly stated what answer ought to be specified. An example of one of these "honeypot" questions is shown in Figure 7. Amazon Mechanical Turk workers got  $83\%$  accuracy on these questions. Expert raters got  $91.8\%$  accuracy. Table 8 shows the accuracy of each expert rater along with the number of annotations they provided. Table 9 shows the example excerpts that were used to "train" the expert raters.
For both the Amazon Mechanical Turk raters and the expert raters initial predictions were biased towards 'possibly human,' and only by observing more tokens did their predictions become more confident. Figure 4 shows that 'possibly human' is by far the most frequent answer upon observing 16 tokens, and as more tokens are observed raters gravitate towards 'definitely human' or 'definitely machine.' Even at 192 tokens, many raters are still uncertain. Figure 4 also shows how raters for the most part default to guessing short excerpts are
Figure 4: Number of votes expert raters made for each label as a function of number of tokens observed. As raters observe more tokens, their predictions become more confident.
human-written, and as the excerpts are extended, raters use the extra evidence available to revise their guess. By the longest sequence length, votes for "human-written" and "machine-generated" are about balanced.
In Figure 5, we plot the frequency for each sequence length that raters converged on a single guess (either human or machine) at that point. The figure shows how it takes raters longer to converge on a decision of "machine" than to converge on a decision of "human."
A.3 Automatic Detection Method Reliability
In order to quantify the variance of automatic discriminator accuracy, we finetuned five independent BERT discriminators on a 'mixed' dataset comprising of  $50\%$  human-written examples and  $50\%$  machine-generated examples, where machine-generated examples are equally split between top- $k = 40$ , top- $p = 0.96$ , and untruncated random sampling. All sequences were exactly 192 tokens. The best performing model checkpoint, according to an in-domain validation set, was then used to evaluate out-of-domain binary classification datasets as in Table 2 of the main paper.
The results are shown in Table 7. We find out-of-domain accuracy to be extremely reliable with a standard deviation of approximately  $1\%$  or less.
Method	# train	# valid	# test
large-744M-k40-1wordcond	211148	4226	4191
large-744M-k40-nocond	218825	4362	4360
large-744M-p0.96-1wordcond	210587	4248	4208
large-744M-p0.96-nocond	209390	4174	4185
large-744M-p1.0-1wordcond	209334	4169	4173
large-744M-p1.0-nocond	208219	4187	4168
human-written	201344	4031	4030
Table 5: The number of excerpts used for training, validation, and testing.
# Annotations	Expert Raters	AMT Workers
webtext	239	450
k0-1wordcond	87	150
k40-1wordcond	75	150
p0.96-1wordcond	74	150
total machine	236	450
Figure 5: On average, it takes much less text for raters to decide an excerpt is human-written than to decide an excerpt is machine-generated.
Table 6: The number of human annotations collected. In total, there were 50 examples from each sampling strategy and 150 examples of web text. Each example was shown to at most three raters.
Dataset	μ	σ
random sampling	72.47	1.02
top-k = 40	88.06	0.59
top-p = 0.96	74.4	0.76
Table 7: Average  $(\mu)$  and standard deviation  $(\sigma)$  of accuracy on out-of-domain datasets across five runs of automatic discriminator finetuning.
Accuracy	Count
61.3%	83
57.8%	51
66.7%	51
69.8%	51
79.5%	48
84.6%	40
82.4%	39
65.6%	36
78.1%	34
84.0%	26
58.8%	18
92.3%	14
90.0%	11
100.0%	9
50.0%	8
60.0%	5
100.0%	5
100.0%	2
0.0%	2
0.0%	1
100.0%	1
0.0%	1
Table 8: Our expert rater pool consisted of 22 raters. The average accuracy of each rater on the longest excerpt length (192 tokens) is shown here along with the total number of excerpts they annotated.
Table 9: The 10 examples that "expert" raters were guided through before they were asked to perform the detection task. These are hand-selected to showcase the spectrum of generated text and human-written text.
Figure 6: The interface of the task used for human evaluation. Each time the user presses next, the passage's length is doubled. On the left, we show the first step of evaluation, on the right, the second to last.
Figure 7: For some of the questions, the text "Dear AMT Worker: to show you're reading, please select definitely [X] for this one." was inserted into the last text segment, and "Did you read carefully?" was appended to the end.