SlowGuess/NAACL_2022_OCR huggingface dataset:
We sampled from the 6.8 million papers in S2ORC which are biomedical[19] to create disjoint datasets for domain-tuning proxy and target GPT-2 generators, for task-tuning RoBERTa-based detectors, and for test sets. Since our attacker model consists of random paragraph replacement, we domain-tuned GPT-2 on 890,000 biomedical paragraphs in order to generate new paragraphs conditioned on previous paragraphs.[20] This is done twice, on non-overlapping data, to obtain two separate generators: the target generator is used to create the test document collections and SME-labeled training paragraph collections, while the proxy generator is used to create proxy training data. The details of each of these datasets are given below.
5.2.1 Test Data
In order to evaluate our detectors against various document tampering scenarios, we use several document-level datasets, which differ by the number of generated (replaced) paragraphs in each document. Each of these test sets consists of 500 human documents and 500 tampered (generated/modified) documents.[21] The five test sets containing tampered documents are given as follows: test-1-fake contains only tampered documents with exactly one synthetic paragraph, text- $x$  replaces every paragraph with a generated paragraph with probability  $x$ , for  $x$  in  $\{0.1, 0.5, 0.9\}$ , and test-all-fake has all paragraphs generated, with each subsequent paragraph generated conditional on the previously generated paragraph. Two examples can be found in Appendix §A.7, Table 13.
$^{14}$ Fine-tuning hyperparameters are shown in §A.1.
15 Available at https://core.ac.uk/services/dataset/
16The list of open access data providers for each STEM subject is provided in Table 10 in Appendix §A.6.
17 In the absence of additional information, it seems reasonable to assume an uninformative prior wherein the classes are balanced. In practice, one could base this prior on the empirical distribution over classes in the SME-labeled data, which approximates the distribution of real and generated samples in the corpus under consideration.
18v. 1 (2020-07-05) at https://github.com/allenai/s2orc.
<sup>19</sup>We selected all papers with a Microsoft Academic Graph field of study which included Biology or Medicine, or which had a PubMed or PMC ID tag (these are biomedical in a broad sense: the  $15\%$  of papers that were not tagged with Biology or Medicine were on the related fields of Chemistry, Psychology, Physics, Math or Computer Science applied to biomedicine.)
$^{20}$ Details on the fine-tuning of GPT-2 are in Appendix §A.1.
21 For each test dataset we use the same 500 human documents and the same 500 documents to tamper with.
5.2.2 Training Data
Here we describe the datasets used to build paragraph-level detectors.
SME-labeled data Nested subsets of 10,000 paragraphs (of sizes  $\{0,100,500,1\mathrm{k},10\mathrm{k}\}$ ) were used as SME-labeled (in-domain) training data, with equal numbers of real and generated paragraphs. The generated paragraphs are obtained using the same domain-tuned generator as for the test sets. For each paper, we sample one paragraph to use as real, and one paragraph to condition on.
Proxy data The proxy GPT-2 generator is used to obtain nested subsets of 10,000 paragraphs (of sizes  $\{0,100,500,1\mathrm{k},10\mathrm{k}\}$ ) in much the same way as for the in-domain data, except for one difference. Unlike for the in-domain data, the defender has access to every human paragraph that is being replaced by a (proxy) generated paragraph[22]. So for each paper, we sample a real paragraph at position  $i$ , and use the previous paragraph at position  $i - 1$  as a prompt for the generated paragraph.
What if the defender uses a proxy generator designed for unconditional paragraph generation? Preliminary experiments on using proxy data generated without prompting showed only a small drop in accuracy under most dataset sizes, so this is not a strong assumption to make.[23]
6.1 Detection of Generated Abstracts
Detection performance on Semantic Scholar abstracts depends on the model used for task-tuning (RoBERTa-base, RoBERTa-large, or RoBERTa-large-STEM), the proxy domain (biomedical, physics, or biomedical-physics), and the number of proxy and SME samples used for task-tuning, with full results shown in Appendix §A.3. We consider the effects of these four dimensions below.
Interplay of SME and proxy labels Some SME labels are always needed for good performance, even when the training proxy text is in a similar domain as the target domain (Table 3). Without any SME labels, the highest accuracy that could be achieved with RoBERTa-large using only biomedical proxy text was .67 (with 10k proxy samples).
Proxy samples
0	100	500	1k	10k	100k
SME samples	0	.59	.58	.54	.67
100	.68	.67	.89	.89	.81
500	.69	.68	.82	.75	.88
1k	.80	.90	.84	.89	.90
10k	.92	.95	.94	.94	.95
Table 3: Detection accuracy when task-tuning RoBERTa-large with biomedical proxy data.
n	Biomedical	Biomed-physics	Physics
100	.67	.87	.78
500	.89	.82	.83
1k	.89	.85	.84
10k	.81	.69	.62
100k	.82	.70	.66
Table 4: Comparing proxy domains when task-tuning RoBERTa-large on  $n$  proxy samples (from different domains) and 100 SME samples.
Task-tuning the 10k-proxy model checkpoint a second time with only 100 SME-labeled samples resulted in a large improvement (accuracy of .81, recall of .95).
Given a fixed number of SME samples, increasing the number of proxy samples improves performance, but only up to a point, after which the proxy data starts being detrimental. For example, given 100 SME examples (row 2 in Table 3), the highest performance for the biomedical proxy experiments was achieved when using between 500 and 1000 proxy samples (resulting in accuracy of .76-.77 for RoBERTa-base, .89 for RoBERTa-large and .91-.92 for RoBERTa-large-STEM). The same observation holds when using physics and biomedical-physics proxy text. Unsurprisingly, the effects are worse with increasing domain shift. The decrease in accuracy when jumping from 1k to 10k proxy samples is .08 for biomedical proxy data, .16 for biomedical-physics proxy data, and .22 for physics proxy data.
Effect of domain shift When 1k or 10k SME labels were available, performance across task-tuning domains was similar. On the other hand, with 500 SME labels or less the impact of the proxy data domain was greater, as shown in Table 4 for the case of 100 SME labels. For most proxy dataset sizes, there is a decrease in performance as one moves from biomedical to physics.
Finally, we note that when using RoBERTa-
22On the other hand, the in-domain data is labeled by a SME, who identifies generated paragraphs but does not have access to the human paragraphs that they replaced.
23See Appendix  $\S \mathbf{A}.5$  for details.
large-STEM, 100 SME samples are sufficient to achieve .91 accuracy when using biomedical proxy samples. However, if the proxy samples come from a different domain (physics, or a mix of physics and biomedicine), then 500 SME samples are required to achieve the same accuracy.
Effect of model size RoBERTa-large had higher accuracy than RoBERTa-base under most scenarios, with an absolute increase in accuracy ranging between 1 and 22 points. RoBERTa-large always had higher precision than RoBERTa-base.
Effect of domain-tuning Pre-training RoBERTa on a diverse corpus of STEM technical text improved performance in most cases, sometimes by a large margin. RoBERTa-large-STEM outperformed RoBERTa-large even when the target and proxy domains were close (i.e., using biomedical proxy data), with 1 to 5 point gains in accuracy under most conditions.
Domain-tuning RoBERTa was more beneficial with increasing amounts of domain shift, and when task-tuning on a large number of proxy samples and a small number of SME-labeled samples, as shown in Figure 3. A domain-specific RoBERTa is better at recovering from being trained on a large volume of data from the wrong domain, when given a small amount of in-domain text.
Figure 3: Increase in accuracy when switching from RoBERTa-large to RoBERTa-large-STEM, when task-tuning on 10k proxy samples.
6.2 Detection of Tampered Documents
The performance of the RoBERTa-large detectors at the paragraph level is shown in Table 5. It is not unrealistic to assume a SME can label 100 paragraphs (50 real, 50 synthetic) to create a detector with .94 accuracy. Thus, in the rest of the section we shall only evaluate the classifier trained on 10k conditioned proxy samples and 100 SME samples.
Identifying documents with at least one synthetic paragraph Here we apply the scoring
SME samples	Proxy samples
0	100	500	1k	10k
0	.60	.78	.84
100	.77	.82	.87	.89
500	.85	.89	.90	.90
1k	.91	.91	.90	.92
10k	.96	.95	.96	.96
Table 5: Accuracy when task-tuning RoBERTa-large on conditioned proxy data, evaluated on a balanced subset of 1000 real and 1000 synthetic paragraphs.
Test set	T=500	T=1000
test-all-fake	.87 [.79, 1.0]	.98 [.96, 1.0]
test-0.9	.87 [.79, 1.0]	.97 [.96, .99]
test-0.5	.86 [.79, .99]	.96 [.96, .96]
test-0.1	.81 [.77, .89]	.80 [.94, .63]
test-1-fake	.76 [.74, .78]	.67 [.91, .37]
Table 6: Performance (accuracy, [precision, recall]) when predicting whether documents contain at least one synthetic paragraph (after removing paragraphs shorter than  $T$  ).
strategies (S1) and (S1-T) described in §4.2 to predict whether a given document has at least one generated paragraph. We found that on all our test sets, scoring documents using (S1) resulted in recall of .99-1 but precision at nearly chance level (.55-.56). This is due to the high false positive rate for short paragraphs. To remedy this, we score using (S1-T), ignoring all paragraphs shorter than a given threshold of  $T$  characters $^{24}$ . The results for  $T = 500$  and  $T = 1000$  are shown in Table 6.
When  $T = 1000$ , performance is high for test-all-fake, test-0.9 and test-0.5; however, recall drops substantially for test-0.1 and test-1-fake. This is due to the fact that these test sets contain very few generated paragraphs, which are then more likely to be filtered out: all synthetic paragraphs were removed from  $64\%$  of tampered test-1-fake documents, and  $39\%$  of tampered test-0.1 documents. Unfortunately, filtering less aggressively with  $T = 500$  improves recall at the cost of lower precision across all test sets.
Ranking documents by fraction of generated content We rank documents using (S2), i.e., according to the estimated fraction of paragraphs clas
24If all paragraphs in a document are shorter than  $T$ , we score the longest three paragraphs, but this is a rare occurrence (less than  $5\%$  of documents when  $T = 1000$ ).
Test set	P@100	P@250	P@500
test-all-fake	1.0	1.0	.99
test-0.9	1.0	1.0	.99
test-0.5	.98	.97	.95
test-0.1	.70	.74	.69
test-1-fake	.57	.60	.59
Table 7: Ranking performance when ranking documents according to the estimated fraction of paragraphs classified as generated.
sified as generated. Table 7 shows the fraction of documents in the top-k ranked documents that were tampered with  $(\mathrm{P}@\mathrm{k})$ . It is possible to retrieve nearly all tampered documents, except for test sets test-0.1 and test-1-fake.
Effect of paragraph splitting errors It is unrealistic to assume that the adversary and the defender use the same paragraph splits. For example, text from a PDF file must first be extracted and split into paragraphs before detection can be done. Even when using a good paragraph splitter, it is unlikely that the paragraph splits would exactly match those used by the generator. To investigate the robustness of detection against paragraph splitting errors we process each test set as follows: each paragraph is sentence-segmented with scispaCy v.0.2.2 (Neumann et al., 2019), and every five consecutive sentences is taken to be a paragraph, disregarding the original paragraph splits. This is a stress-test for our detectors, since one can probably achieve far fewer paragraph-splitting errors when using a paragraph splitter such as GROBID (Lopez, 2009).
When applying the detector on the incorrectly-split documents and scoring with (S1), we find that precision increases from .55 to between .61 and .67. This is due the fact that paragraphs in the human 5-sentence splits are generally longer than the original paragraphs. On the other hand, recall drops significantly for test-0.1 and test-1-fake (from .99 to .87, and .99 to .79, respectively). This is due to the fact that it is harder to detect paragraphs containing a mix of real and synthetic content: recall was .95, .66 and .33 for the the subsets of paragraphs with 5, 4, and 3 synthetic sentences, respectively[25].
In this paper, we studied the problem of automatic detection of GPT-2-generated technical text. We found that RoBERTa-based detectors can be successfully adapted from one scientific discipline (physics) to another (biomedicine), requiring relatively small amounts of in-domain labeled data. These could be provided by a subject matter expert (SME) in a reasonable amount of time. We also evaluated these paragraph-level detectors on a document tampering task, assuming that the adversary replaces randomly-selected paragraphs in a document with generated ones.
Future work should evaluate the extent to which this methodology would work on detecting text from newer generators such as GPT-3. Other challenging scenarios include adding noisy labels (e.g., if a SME makes a certain fraction of mistakes), and class imbalance. Our results on document tampering (i.e., that it is significantly harder to detect small amounts of generated text intermingled amongst real text) also point to the need to develop detection pipelines for other threat models such as single word or phrase substitutions (Schuster et al., 2020). As text generation techniques continue to improve, it may be that more interpretable, fact-checking approaches are required to detect both human and machine-generated misinformation.
We would like to thank Greg Durrett, Katrin Erk, Jessy Li, and the anonymous reviewers for their helpful comments and suggestions.
Improvements in the detection of synthetic text could be used by an adversary to improve the quality of generated text or to help them avoid detection (Darmetko, 2021). False positives are another source of potential negative consequences of automated detectors. For example, incorrectly flagged human-written content could be a source of misinformation, and could additionally lead to a loss of trust in the detection system. Care should be taken that false positives do not affect certain demographics disproportionately (Bommasani et al., 2021, §5.2). Finally, widespread awareness of the mere possibility of synthetic scientific text can further undermine public trust in genuine science (Makri, 2017).
25This affected test-0.1 and test-1-fake the most, because these test sets had a greater fraction of paragraphs with mostly human sentences but with some generated sentences mixed in.
David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human- and machine-based detection. In Advanced Information Networking and Applications - Proceedings of the 34th International Conference on Advanced Information Networking and Applications, AINA-2020, Caserta, Italy, 15-17 April, volume 1151 of Advances in Intelligent Systems and Computing, pages 1341-1354. Springer.
Nadisha-Marie Aliman and Leon Kester. 2021. Epistemic defenses against scientific and empirical adversarial AI attacks. In CEUR Workshop Proceedings, 2021 Workshop on Artificial Intelligence Safety, AISafety 2021, 19 August 2021 through 20 August 2021. CEUR-WS.
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elghohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. Construction of the literature graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 84-91, New Orleans - Louisiana. Association for Computational Linguistics.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc'Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? Learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351.
Alberto Bartoli and Eric Medvet. 2020. Exploring the potential of GPT-2 for generating fake reviews of research papers. In *Fuzzy Systems and Data Mining* VI: Proceedings of FSDM 2020, volume 331, pages 390–396. IOS Press.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Anja Belz. 2019. Fully automatic journalism: We need to talk about nonfake news generation. In Proceedings of the 2019 Truth and Trust Online Conference (TTO 2019), London, UK, October 4-5, 2019.
Meghana Moorthy Bhat and Srinivasan Parthasarathy. 2020. How effectively can machines defend against machine-generated fake news? an empirical study.
In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 48-53, Online. Association for Computational Linguistics.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
Guillaume Cabanac and Cyril Labbe. 2021. Prevalence of nonsensical algorithmically generated papers in the scientific literature. Journal of the Association for Information Science and Technology.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Hahuong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics.
Evan Crothers, Nathalie Japkowicz, Herna Viktor, and Paula Branco. 2022. Adversarial robustness of neural-statistical features in detection of generative transformers. arXiv preprint arXiv:2203.07983.
Tomasz Darmetko. 2021. Fake or not? generating adversarial examples from language models. Undergraduate thesis, Maastricht University.
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492-2501, Online. Association for Computational Linguistics.
Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.01294.
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021. TweepFake: About detecting deepfake tweets. PLoS ONE, 16(5):e0251415.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Leon Frohling and Arkaitz Zubiaga. 2021. Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover. PeerJ Computer Science, 7:e443.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111-116, Florence, Italy. Association for Computational Linguistics.
Vivian Emily Gunser, Steffen Gottschling, Birgit Brucker, Sandra Richter, and Peter Gerjets. 2021. Can users distinguish narrative texts written by an artificial intelligence writing tool from purely human text? In International Conference on Human-Computer Interaction, pages 520-527. Springer.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238-4248, Hong Kong, China. Association for Computational Linguistics.
Fouzi Harrag, Maria Dabbah, Kareem Darwish, and Ahmed Abdelali. 2020. Bert transformer model for detecting Arabic GPT2 auto-generated tweets. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 207-214, Barcelona, Spain (Online). Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Dirk Hovy. 2016. The enemy in your own camp: How well can we detect statistically-generated fake reviews – an adversarial study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 351–356, Berlin, Germany. Association for Computational Linguistics.