SlowGuess/NAACL_2022_OCR huggingface dataset:
Juan Diego Rodriguez $^{1}$  Todd Hay $^{2}$  David Gros $^{3}$  Zain Shamsi $^{1}$  Ravi Srinivasan $^{1}$
$^{1}$ Applied Research Laboratories, The University of Texas at Austin  $^{2}$ En Solución
<sup>3</sup>University of California, Davis
{juand-r，zain.shamsi，ravi.srinivasan}@utexas.edu
todd.hay@ensolucion.com
dgros@ucdavis.edu
Machine-generated text presents a potential threat not only to the public sphere, but also to the scientific enterprise, whereby genuine research is undermined by convincing, synthetic text. In this paper we examine the problem of detecting GPT-2-generated technical research text. We first consider the realistic scenario where the defender does not have full information about the adversary's text generation pipeline, but is able to label small amounts of in-domain genuine and synthetic text in order to adapt to the target distribution. Even in the extreme scenario of adapting a physics-domain detector to a biomedical detector, we find that only a few hundred labels are sufficient for good performance. Finally, we show that paragraph-level detectors can be used to detect the tampering of full-length documents under a variety of threat models.
Recent advances in techniques for generating realistic synthetic content (i.e., deepfakes) pose a diverse set of problems with significant societal consequences (Kreps et al., 2020; Bommasani et al., 2021; McGuffie and Newhouse, 2020). The advent of large language models for text generation (Radford et al., 2019; Brown et al., 2020) have made it easier than ever to create convincing synthetic<sup>1</sup> text (Solaiman et al., 2019). While much attention has been focused on the role of synthetic audio and video, it can be argued that synthetic text may give rise to some of the most serious threats to information integrity and the long-term preservation of archival knowledge (Aliman and Kester, 2021).
Review of Characterizing Tamarobin System Diseases in Lymphoblastic Cancer Patients
Using the ZiiTELMA monoclonal antibody system (ZezM), we evaluated the natural antigens of tamarobin broth, dextrose, and tamarobin vin B, sodium-citrate. Despite their excellent clinical properties, none of them have antiviral roles in the immune system...
The prefrontal "cortex" in the pigeon catecholamine histofluorescence
The prefrontal cortex of mammals is densely innervated with dopaminergic fibers. We report a comparable, dense network of catecholamine (probably dopamine)-containing fluorescent fibers in the posterodorsolateral neostriatum of the pigeon. This region is clearly separable...
Figure 1: Machine-generated text enables the corruption of technical knowledge (e.g., biomedical research). One of the above abstracts was generated by GPT-2. More examples of generated text can be found in §A.7.
While there are currently no documented cases of published papers containing text from neural language models, non-neural machine-generated papers have already been published in peer-reviewed journals (Cabanac and Labbe, 2021). With more convincing text generation techniques and a growing number of publications, this is a problem that could get much worse, particularly for non-peer-reviewed technical text (Ranade et al., 2021). In addition to hindering the scientific process, synthetic technical texts could be used to manipulate public opinion and sow discord around specific scientific topics (Aliman and Kester, 2021). Other consequences of machine-generated technical text include the contamination of NLP pipelines (Ranade et al., 2021) and poisoning of language models (Schuster et al., 2021).
The rise of realistic machine-generated text – and
<sup>1</sup>Some authors use the term fake text; however, we prefer synthetic, since human-written text can contain false information, while machine-generated text can (sometimes) be factually correct (Schuster et al., 2020; Belz, 2019).
Human Language Technologies, pages 1213 - 1233
its possible misuse – has spurred the development of automated tools to distinguish between genuine and synthetic English text.2 However, most work to date has focused on web text or news, rather than on domain-specific technical text (Solaiman et al., 2019; Zellers et al., 2019). While synthetic technical text is likely to be distinguished by subject matter experts (SMEs), it is difficult for non-domain evaluators to do so. Therefore, given the ease at which large amounts of synthetic text can be generated, automated countermeasures are necessary in order to alleviate the burden on SMEs.
We focus on developing capabilities for detecting generated technical English text and delineating the contexts in which these detection approaches can be applied. We assume an adversary generating technical text using GPT-2 (Radford et al., 2019), but do not know apriori in exactly which technical area. Realistically, some domain shift will be inevitable, which leads us to investigate whether detectors can be applied across domains, e.g., can a detector of generated physics papers be adapted to also detect biomedical text? In addition, most work to date has focused on detecting short pieces of text. We show that automated detection of tampered full-length research papers is possible under various threat models. Our work makes two main contributions:
- We show that accurate cross-domain detection of generated technical text is possible using a small number of in-domain samples and quantify the amount of SME effort required.
- We study the detectability of tampered full-text technical papers (where a subset of paragraphs have been replaced with generated text) under various scenarios.
Automated detection of synthetic text Automated approaches to detecting machine-generated text have included energy-based models (Bakhtin et al., 2019), repurposing the generator as a discriminator (Zellers et al., 2019), and various neural and non-neural classifiers (Solaiman et al., 2019; Ippolito et al., 2020; Uchendu et al., 2020; Zhong et al., 2020; Frohling and Zubiaga, 2021; Fagni
et al., 2021). Most work on automated detection targeted GPT-2-generated text, although Bakhtin et al. (2019), Uchendu et al. (2020), Fagni et al. (2021) and Stiff and Johansson (2021) also experimented with other generators. Finally, there is a growing body of work on the adversarial robustness of automated detectors of synthetic text (Wolff, 2020; Bhat and Parthasarathy, 2020; Stiff and Johansson, 2021; Crothers et al., 2022). A survey on the automatic detection of synthetic text can be found in (Jawahar et al., 2020).
Prior work closest to our study are (Solaiman et al., 2019), (Ippolito et al., 2020), (Munir et al., 2021), (Bakhtin et al., 2019) and (Stiff and Johansson, 2021), which look at cross-domain settings where the distribution of synthetic text used to train a detector differs from the target distribution. The shift could be due to different model architectures (Bakhtin et al., 2019; Stiff and Johansson, 2021), different model sizes (Solaiman et al., 2019), different decoding strategies (Solaiman et al., 2019; Ippolito et al., 2020), or different fine-tuning datasets (Bakhtin et al., 2019; Munir et al., 2021; Stiff and Johansson, 2021). Bakhtin et al. (2019) show that energy-based models generalize poorly across corpora (Wikipedia, news and books), but that training on the union of the source and target domains is effective. Munir et al. (2021) show that XLNet can accurately attribute synthetic text even when the GPT-2-generated portions of the training and test sets come from GPT-2 fine-tuned on different subreddits.
Unlike the previous papers, we evaluate our detectors on technical (biomedical) text, and vary the number of labeled samples available for training in the target domain, under the assumption that source labels are plentiful but target examples are more expensive to obtain (i.e., source samples can be generated at will, but target examples need to be discovered and verified by a human since the adversary's target text generator is generally unavailable). In addition, we evaluate the detection of full-length tampered documents consisting of a mix of real and machine-generated content.
Attribution for synthetic text In addition to distinguishing between real and generated text, one may also wish to determine which system generated a given text (e.g., model type, size, decoding strategy). Variations of this "authorship attribution" problem have been explored by Uchendu et al. (2020), Tay et al. (2020) and Munir et al. (2021).
$^{2}$ Most work on the detection of GPT-2-generated text has focused on English. However, (Harrag et al., 2020) examines automated detection of synthetic tweets in Arabic.  
 $^{3}$ Our code is available at https://github.com/ciads-ut/cross-domain-detection-gpt-2
These works have found that, in general, while the attribution problem is harder than the detection problem, details of the text generator can often be learned from the generated text.
Human detection of synthetic text Human evaluations on the detection of generated text have been conducted for news (Zellers et al., 2019; Brown et al., 2020; Kreps et al., 2020; Clark et al., 2021), product reviews (Hovy, 2016; Yao et al., 2017; Adelani et al., 2020), web text (Gehrmann et al., 2019; Ippolito et al., 2020), stories (Clark et al., 2021; Donahue et al., 2020; Clark et al., 2021; Gunser et al., 2021), peer reviews (Bartoli and Medvet, 2020), cybersecurity text (Ranade et al., 2021) and submissions to federal public comment websites (Weiss, 2019). These studies have shown that it is difficult for people to distinguish between real and neural-generated text. Gehrmann et al. (2019) developed a tool, GLTR, to help users visualize statistical artifacts in generated text.
3 Threat model and defender capabilities
Given that people have trouble distinguishing real (human-written) from GPT-2-generated text (Zellers et al., 2019; Kreps et al., 2020; Clark et al., 2021), we study the detection of generated text under the threat model where an adversary generates text from GPT-2 domain-tuned on domain-specific text. We do not necessarily assume that text from this domain is publicly available in significant quantities. Throughout, we shall refer to those developing the generated text as the adversary, and those building automated detectors of synthetic text as the defender. As there does not (yet) exist a "real-world" dataset of GPT-2 generated technical text, we simulate both the adversary and the defender.[5]
We study two scenarios: where the adversary generates single technical abstracts (§4.1), and where the adversary replaces randomly selected paragraphs in full-length documents with GPT-2-generated paragraphs (§4.2). Other threat models are possible, such as replacing single words, sentences or phrases (Schuster et al., 2020; Donahue
Figure 2: Experimental setup for detecting synthetic text used in this paper. Since the defender does not have access to the adversary's text generator, they develop a proxy generator, in order to mimic the adversary's generator as closely as possible. Proxy and target corpora will generally come from different domains.
et al., 2020; Bhat and Parthasarathy, 2020), and should be studied in future work.
3.2 Defender Capabilities
In general, the defender has little or no information about the adversary's text generation pipeline, including the data used for training, the model architecture used, or the decoding strategy employed. We assume the defender has access to a small number of examples from the adversary, which can be labeled by a subject matter expert (SME) as real (human) or synthetic (generated). In addition, since the defender does not have access to the adversary's target generator, they can build a proxy generator, trained on proxy text, in order to obtain
<sup>6</sup>We assume that a SME is able to accurately carry out this task. In reality, even a SME could make mistakes, but we leave the scenario of noisy labels to future work.
$^{4}$ Following (Han and Eisenstein, 2019), we use domain-tuning to refer to further self-supervised training of a model on unlabeled text in a specific domain, and task-tuning to refer to supervised fine-tuning of a model for a given labeling task.
5More recently, a corpus of GPT-2 generated (NLP-related) abstracts has been created for testing detection methods (Liyanage et al., 2022).
more samples of real and synthetic text which are (hopefully) statistically similar to the ones obtained from the adversary<sup>7</sup>, as shown in Figure 2. The defender can then train a model to detect generated text using both the in-domain SME-labeled text and the (possibly) out-of-domain proxy text, as described in more detail in §4. The level of access the defender is assumed to have to various parts of the adversary generation pipeline is shown in Table 1.
Defender's access level
Adversary's (target) generator model	None
Labeled samples from adversary domain	Limited (SME)
Real samples from proxy domain	Plentiful
Synthetic samples from proxy domain	Unlimited
Table 1: Assumptions about the defender's access to data or models that can be used in a detection pipeline. We assume a SME is able to label a small (random) sample consisting of real and adversary-generated text.
Since the effect of differing model sizes has already been explored (Zellers et al., 2019; Solaiman et al., 2019; Frohling and Zubiaga, 2021), we assume the adversary and defender use the same sized model (GPT-2 Medium, with 355M parameters), but domain-tuned on different datasets. Since the defender cannot guess the temperature value used by the target generator, we shall decode using a temperature value of 1.0 for the target generator, but 0.8 for the proxy generator. We also assume both adversary and defender use nucleus sampling (Holtzman et al., 2020). The defender would not know whether the generator is using nucleus sampling. However, it was shown in (Ippolito et al., 2020; Solaiman et al., 2019) that a discriminator trained with nucleus sampling is able to perform nearly as well at detecting text generated with top-k (Fan et al., 2018) sampling as with nucleus sampling, while a discriminator trained using top-k sampling fails to detect generations from nucleus sampling. Hence this is a reasonable assumption to make.
Our experiments will investigate the effects of domain shift between the proxy and target data, as well as the number of SME-labeled examples available to the defender. While the assumption that
both adversary and defender share the same model architecture may seem like a strong one, we fix the architecture so as to focus on the effect of domain shift, and note that previous works have already studied the effect of using different architectures (Bakhtin et al., 2019; Stiff and Johansson, 2021), model sizes (Solaiman et al., 2019), and decoding strategies (Ippolito et al., 2020; Solaiman et al., 2019).
4 Automated Detection
4.1 Detecting Generated Abstracts
While the defender could train a detector on the union of SME-labeled (target) and proxy data as done in (Zellers et al., 2019), we instead follow a pipelined approach by first task-tuning on the proxy real and synthetic text to produce a proxy detector, before task-tuning a second time on the SME-labeled text. One advantage of this approach is that one would still have a detector even when SME labels are not available. In addition, previous studies have shown that task-tuning twice can yield good performance on a variety of tasks and can help mitigate the effects of domain shift (Phang et al., 2018; Sellam et al., 2020).
Preliminary experiments on in-domain detection showed that fine-tuning RoBERTa consistently outperformed other classifiers (LSTMs, HAN, BERT and XLNet), as shown in Appendix §A.2, and so we exclusively use RoBERTa in all of our experiments. This is in line with results in (Sollaiman et al., 2019; Uchendu et al., 2020; Fagni et al., 2021), which showed the effectiveness of RoBERTa for detecting GPT-2-generated text. In order to investigate the benefit of further pretraining (Gururangan et al., 2020), we also domain-tune RoBERTa on technical text from various science, technology, engineering, and mathematics (STEM) fields, as described in §5.1.2; we call the resulting model RoBERTa-large-STEM. Our experiments will vary both the number of proxy and SME-labeled abstracts, as well as the subject domain of proxy text, as detailed in §5.1.
4.2 Detecting Tampered Documents
We also investigate how well our detection methods work when applied at the document level, assuming the following attacker model: a fraction of randomly-selected paragraphs in a document are replaced by generated paragraphs (see §5.2.1 for details). Each generated paragraph is conditioned
<sup>7</sup>If text from the target domain is available to the defender in large quantities, the proxy domain is effectively the same as the adversary (target) domain.
The effect of differing temperatures is also explored in (Munir et al., 2021).
Indeed, viewing the accuracy scores in Table 2 of (Ippolito et al., 2020) as the payoff to the defender in a zero-sum game, nucleus sampling is the minimax strategy for both defender and generator.
on the previous paragraph (i.e., the previous paragraph is used as a prompt). Conditioning helps the text stay on-topic, and would likely help evade coherence-based detectors $^{10}$  (Singla et al., 2021). We refer to the modified documents as tampered rather than generated, since only a subset of the document might be generated.
On the detection side, we train paragraph-level detectors, and then aggregate paragraph scores into document scores to classify documents. We need to specify both the question we would like to answer, as well as how to aggregate the detectors' paragraph scores  $\{s_i\}$  to answer it. Here we interpret  $s_i$  as the probability that a paragraph has been generated. The general question "has this document been tampered with?" leads to two more specific questions and associated scoring strategies:
- (S1) Is at least one paragraph in the document generated? The probability that a document with paragraph scores  $\{s_i\}$  has at least one synthetic paragraph is then given by:
$$
P = 1 - \prod (1 - s _ {i})
$$
- (S2) What fraction of a document is generated? For a document with  $N$  paragraphs, this is:
$$
F = \frac {1}{N} \sum \mathbb {1} _ {s _ {i} > 0. 5}
$$
As we will see, one drawback of scoring with (S1) is that it is extremely sensitive to false positives. Entirely human-written documents have a high chance of having one or two false positives (especially among short paragraphs), even if the other paragraphs are correctly classified. In these cases the human documents will be classified as tampered. Since the false positive rate is highest for short paragraphs<sup>11</sup>, we can filter out very short paragraphs from each document before scoring. We thus also experiment with the following score:
- (S1-T) Is at least one paragraph  $p_i$  longer than a given threshold  $T$  synthetic? The probability that this is the case is:
$$
P = 1 - \prod \left(1 - s _ {i} \cdot \mathbb {1} _ {\operatorname {l e n} \left(p _ {i}\right) > T}\right)
$$
For each document we can use thresholds for  $P$  or  $F$  to decide how to classify the document; in addition, we shall use (S2) to rank documents by how much generated content they contain.
Here we describe the datasets used and other experimental details when simulating the adversary and defender's pipelines (Figure 2). The experiments on synthetic abstract detection are described in §5.1; experiments on tampered document detection are described in §5.2.
5.1 Real and Synthetic Abstracts
Table 2 summarizes the datasets used. The adversary has access to the Semantic Scholar Open Research Corpus $^{12}$  (Ammar et al., 2018), while the defender has access so several subsets of CORE (Knoth and Zdrahal, 2012), as well as small amount of text labeled by a SME. We chose abstracts from CORE related to biomedicine and physics, since these subjects had the largest number of abstracts for fine-tuning models (shown in §A.6, Table 10).
Synthetic abstract detection
Corpus	Purpose
Semantic Scholar (Biomed+CS)	• Real portion of the test set
• Domain-tune adversary's generator
• Task-tune detector on real (“SME-labeled”) samples from adversary domain
CORE (Biomed, Physics, Biomed+Physics)	• Domain-tune proxy generator
• Task-tune detector on proxy data
CORE (STEM): (all subjects listed in §A.6)	• Domain-tune RoBERTa on STEM text (“RoBERTa-large-STEM”)
Table 2: List of datasets and how each was used for the cross-domain synthetic abstract detection experiments. The Semantic Scholar Open Research Corpus is used for the adversary's generation pipeline, while subsets of CORE are used for the defender's pipeline.
5.1.1 Test Data
We first discuss the construction of datasets used to evaluate the detection of GPT-2-generated abstracts. The test set consists of 1000 real and 1000 synthetic abstracts, generated using GPT-2 domain-tuned on abstracts from the January 2019 version of the Semantic Scholar Open Research Corpus (Ammar et al., 2018) $^{13}$ , hereafter referred to as
<sup>10</sup>Such detectors would use paragraphs in context, rather than independently, and should be considered in future work.
11The effect of paragraph length on the false positive rate is shown in Figure 7 in Appendix §A.4. Similar results can also be found in (Ippolito et al., 2020; Munir et al., 2021).
12These abstracts are mostly, but not entirely, biomedical (Beltagy et al., 2019).
<sup>13</sup>From https://api_semanticscholar.org/corpus/download/
Semantic Scholar. $^{14}$  Text was generated using nucleus sampling, with  $p$  sampled uniformly between 0.9 and 1.0, since (Zellers et al., 2019) showed detection was hardest with  $p$  in that range.
5.1.2 Training Data
Here we describe the SME-labeled abstracts (in-domain) and proxy (possibly out-of-domain) data used to train detectors of GPT-2 generated abstracts.
SME-labeled data Nested subsets of another 10k abstracts (of sizes {100, 500, 1000, 1k, 10k}), obtained in the same way as the test set, were used as "SME-labeled" (in-domain) data for training.
Proxy data Out-of-domain abstracts were sampled from the CORE dataset of open access research papers (Knoth and Zdrahal, 2012), version 2018-03-01 $^{15}$ . CORE covers a wide variety of subjects, some of which can be identified from each paper's data provider. We used a subset of CORE related to STEM fields $^{16}$ , and filtered out non-English abstracts using the Python package langdetect. This subset of CORE was also used to domain-tune RoBERTa-large on STEM abstracts, resulting in RoBERTa-large-STEM.
We trained proxy generators using the biomedical and physics portions of CORE (237,620 abstracts for each), and on the union of the biomedical and physics portions ("biomedical-physics", with 475,240 abstracts). For each of these three generators, we created nested subsets (of sizes \{100, 500, 1000, 1k, 10k, 100k\}) of proxy training data. As with the SME-labeled data, half of the samples were real and half were generated<sup>17</sup>. In the case of biomedical-physics, we used equal numbers of real biomedical and real physics text. We estimate that roughly half of the biomedical-physics generations were biomedical. Examples of generated physics and biomedical abstracts can be found in Appendix §A.7.
5.2 Real and Tampered Documents
While the CORE corpus includes full-length documents, they have not been pre-processed and are rather noisy. Fortunately, the S2ORC corpus (Lo et al., 2020) $^{18}$  includes millions of pre-processed scientific documents. The full-length papers in S2ORC have been preprocessed with paragraph splitting; in addition, captions, tables, headers, footers, footnotes, abstracts and bibliography have been removed from the main text.