SlowGuess/NAACL_2022_OCR huggingface dataset:
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808-1822, Online. Association for Computational Linguistics.
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296-2309, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Petr Knoth and Zdenek Zdrahal. 2012. CORE: three access levels to underpin open access. D-Lib Magazine, 18(11/12):1-13.
Sarah Kreps, R Miles McCain, and Miles Brundage. 2020. All the news that's fit to fabricate: AI-generated text as a tool of media misinformation. Journal of Experimental Political Science.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Vijini Liyanage, Davide Buscaldi, and Adeline Nazarenko. 2022. A benchmark corpus for the detection of automatically generated text in academic publications. arXiv preprint arXiv:2202.02013.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.
Patrice Lopez. 2009. GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International conference on theory and practice of digital libraries, pages 473-474. Springer.
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. 2019. Towards generalizable neuro-symbolic systems for commonsense question answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 22-32, Hong Kong, China. Association for Computational Linguistics.
Anita Makri. 2017. Give the public the tools to trust scientists. Nature News, 541(7637):261.
Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of GPT-3 and advanced neural language models. Technical report, Center on Terrorism, Extremism, and Counterterrorism, Middlebury Institute of International Studies at Monterrey.
Shaoor Munir, Brishna Batool, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2021. Through the looking glass: Learning to attribute synthetic text generated by language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1811-1822, Online. Association for Computational Linguistics.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Priyanka Ranade, Aritran Piplai, Sudip Mittal, Anupam Joshi, and Tim Finin. 2021. Generating fake cyber threat intelligence using transformer-based models. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pages 1-9. IEEE.
Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), pages 1559-1575. USENIX Association.
Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. 2020. The limitations of stylometry for detecting machine-generated fake news. Computational Linguistics, 46(2):499-510.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Yaman Kumar Singla, Swapnil Parekh, Somesh Singh, Junyi Jessy Li, Rajiv Ratn Shah, and Changyou Chen. 2021. AES are both overstable and oversensitive: Explaining why and proposing defenses. arXiv preprint arXiv:2109.11728.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
Harald Stiff and Fredrik Johansson. 2021. Detecting computer-generated disinformation. International Journal of Data Science and Analytics, pages 1-21.
Yi Tay, Dara Bahri, Che Zheng, Clifford Brunk, Donald Metzler, and Andrew Tomkins. 2020. Reverse engineering configurations of neural text generation models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 275-279, Online. Association for Computational Linguistics.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384-8395, Online. Association for Computational Linguistics.
Max Weiss. 2019. Deepfake bot submissions to federal public comment websites cannot be distinguished from human submissions. Technology Science.
Max Wolff. 2020. Attacking neural text detectors. In Towards Trustworthy ML: Rethinking Security and Privacy for ML Workshop, Eighth International Conference on Learning Representations, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Yuanshun Yao, Bimal Viswanath, Jenna Cryan, Haitao Zheng, and Ben Y. Zhao. 2017. Automated crowd-turfing attacks and defenses in online review systems. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017, pages 1143-1158. ACM.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada.
Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Neural deepfake detection with factual structure of text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2461-2470, Online. Association for Computational Linguistics.
A.1 Fine-tuning hyperparameters
Here we list the main hyperparameters used in our experiments.
Domain-tuning GPT-2 to generate abstracts The GPT-2 generators (both proxy and target generators) were domain-tuned with a block size of 512 BPE tokens, and a batch size of 6 on two Titan RTX GPUs, with the Adam optimizer.
Domain-tuning GPT-2 to generate paragraphs in context Proxy and target GPT-2 generators were obtained by domain-tuning GPT-2 with a block size of 768 and a batch size of 3 for four epochs on two disjoint, random subsets of S2ORC biomedical papers, each consisting in about 890,000 paragraphs.
We formatted the fine-tuning data in order to generate complete paragraphs conditioned on previous paragraphs; i.e., each training instance consisted of text from consecutive paragraph pairs: the last 256 tokens of a paragraph, a special newline token, and the next paragraph. For each paragraph pair  $(A,B)$ , truncating  $A$  at 256 tokens allowed us to encode at least 512 tokens for each paragraph  $B$ , and about  $95\%$  of paragraphs in the domain-tuning dataset are shorter than 512 tokens. Thus, the model could learn how to end paragraphs naturally.
Task-tuning RoBERTa We task-tuned all RoBERTa models with a block size of 512 on two Titan RTX GPUs. For RoBERTa-base, we used a batch size of 40, while for RoBERTa-large and RoBERTa-large-STEM we used a batch size of 7. Preliminary experiments suggested that for the smaller task-tuning datasets training for more epochs improved performance. The 100-sample task-tuning dataset was trained for 80 epochs, the 500-sample dataset was trained for 16 epochs, and the other datasets were trained for 8 epochs. Task-tuning on the SME-labeled (target) text was done using the same hyperparameters as were used for the proxy task-tuning.
A.2 Comparison of classifiers on in-domain detection
Table 8 compares several classifiers on the indomain detection task (i.e., the real portions of the train and test sets are from the Semantic Scholar corpus, and the synthetic portions were produced by the same GPT-2-Medium generator).
RoBERTa-large (Liu et al., 2019) and XLNet-large (Yang et al., 2019) outperform the other classifiers. We chose RoBERTa over XLNet because XLNet is known to be unstable when task-tuned on small datasets (Ma et al., 2019).
Discriminator	Accuracy
LR (BOW)	.64
LSTM	.67
HAN	.72
BERT-base	.86
BERT-large	.90
XLNet-base	.89
XLNet-large	.95
RoBERTa-base	.93
RoBERTa-large	.95
Table 8: Detection accuracy when for a variety of models trained on 10,000 in-domain abstracts.
LR (BOW) indicates logistic regression with unigram count features, and HAN (Hierarchical Attention Network) is a hierarchical LSTM with two attention layers, one for words and another for sentences (Yang et al., 2016).
Both the LSTM and the HAN used pre-trained GloVe embeddings $^{26}$  (Pennington et al., 2014). They were trained for 60 epochs with early stopping (patience 10), with 128 hidden layer units, dropout of 0.5, a batch size of 64, learning rate of 0.001 and the Adam optimizer. For the HAN, abstracts were truncated at the first 20 sentences and only the first 50 tokens in each sentence were used. For the LSTM, abstracts were truncated at 200 tokens.
A.3 Full cross-domain results
Figures 4, 5 and 6 contain the full results (accuracy, precision and recall) for all the cross-domain experiments discussed in §6.1.
$^{26}$ https://nlp.stanford.edu/data/glove.42B.300d.zip
Figure 4: Detection performance when using biomedical proxy data.
Figure 5: Detection performance when using physics proxy data.
A.4 Effect of paragraph length on detection performance
There is a higher percentage of short paragraphs in the body of scientific papers than there is among abstracts. We noticed this can lead to difficulties in detection, since detector performance deteriorates with shorter text lengths (Ippolito et al., 2020; Munir et al., 2021).
To investigate the effect of paragraph length on performance, we ranked the human paragraphs by their length in characters and binned them (200 per bin), in order to calculate the false positive rate within each bin. The false negative rates were calculated similarly using the generated paragraphs. These are shown in Figures 7 and 8. The false positive and false negative rates can be seen to increase substantially for paragraphs with less than 500 characters.
Figure 6: Detection performance when using biomedical and physics proxy data.
Figure 7: False positive rate as a function of paragraph length.
A.5 Effect of training on paragraphs generated without conditioning
Since it might be unrealistic to assume that both the adversary's (target) generator and the defender's (proxy) generator both generate text in the same way – by conditioning on the last 256 tokens of the previous paragraph – we verified that our results are not heavily dependent on this assumption by also testing detection performance using unconditional (i.e., unprompted) synthetic paragraphs as proxy training data.
Tables 5 and 9 show paragraph-level accuracy when training on conditioned and unconditioned proxy data, respectively. As expected (Tay et al., 2020), performance is higher when training with conditioned proxy generations than with unconditioned proxy generations if no SME samples are used (e.g., an increase in .10 in accuracy when using  $10\mathrm{k}$  proxy samples). This is mostly due to a large increase in recall (up to .21 for  $10\mathrm{k}$  proxy samples). However, a second round of task-tuning with
Figure 8: False negative rate as a function of paragraph length.
Proxy samples
0	100	500	1k	10k
SME samples	0	.51	.73	.78
100	.77	.75	.83	.88
500	.85	.83	.89	.94
1k	.91	.92	.88	.92
10k	.96	.95	.96	.96
Table 9: Accuracy when task-tuning RoBERTa-large on unconditioned proxy data, evaluated on a balanced subset of 1000 real and 1000 synthetic paragraphs.
SME samples helps close the gap between task-tuning with conditioned and unconditioned proxy samples. Indeed, 100 SME samples are enough to nearly close the gap (a difference of .02 in accuracy).
A.6 STEM subset of the CORE dataset
Subject	%	Open access data providers
Biomedical	28.1	PubMed Central
Nature Precedings
Publications from Karolinska Institutet
Collection Of Biostatistics Research Archive (COBRA)
e-publications@RCSI (Royal College of Surgeons in Ireland)
DigitalCommons@TMC (Texas Medical Center)
Digital Commons@Becker (Washington University School of Medicine)
Physics	39.8	CERN Document Server
arxiv (astro-ph, cond-mat, gr-qc, hep, nlin-chao-dyn, nucl, quant-ph, physics)
Computer Science	3.4	arxiv (cs)
Dagstuhl Research Online Publication Server
CiteSeerX
Computer Science Technical Reports at Virginia Tech
Mathematics	6.1	arxiv (math)
University of Oxford Mathematical Institute Eprints Archive
NUMDAM
Bulgarian Digital Mathematics Library at IMI-BAS
Various subjects	22.6	Naval Postgraduate School
University of Oxford Mathematical Institute Eprints Archive
Massachusetts Institute of Technology
California Institute of Technology
Imperial College London
National University of Singapore
HAL-Polytechnique
Thèses en Ligne (TEL)
Universitat Politècnica de Catalunya
University of Strathclyde
Digital library of Brno University of Technology
Total	100
Table 10: Subjects, data providers, and fraction of (English-language) abstracts per subject in the STEM subset of CORE used to domain-tune RoBERTa-large-STEM. This subset contains 916,074 abstracts. The Biomedical portion (237,620 abstracts), and a subset of the Physics portion (237,620 abstracts) were used to domain-tune GPT-2 to generate physics and biomedical abstracts.
A.7 Examples of generated technical text
Tables 11 and 12 show examples of GPT-2-generated biomedical and physics abstracts. Table 13 shows two examples of tampered documents, where some of the paragraphs were replaced by GPT-2-generated paragraphs. Some of the (human-detectable) errors in the generated abstracts are given in Appendix §A.8.
Generated biomedical abstracts
Cognitive health remains a widely misunderstood area of dementia research. Findings from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) continue to indicate that people with dementia have varying degrees of personality disorders, one of which is asocial functioning. Given that individuals with dementia frequently experience emotions and cognitions such as anger and blame, it is important to understand the factors that contribute to this range of dysfunctions in cognitive functions. To address this issue, we review current thinking regarding how people with dementia and their caregivers understand these issues. We describe the functional distinction between healthy and low-function functioning dementia, discuss the relationship between dementia and temperament, and suggest future directions for future research and development.
Finds from modern groundwater samples have helped to nail down the mineralogy of many deposits of accessible minerals in the Northwest Passage, including Baker Fountain, upper Boundary Haven, and Meishinmaru Harbor. Belying all this, however, have been the potential variability of minerals in this area that are often overlooked, especially the Mocharla Complex, which contains all of the elements of known elements within only  $4\%$  of metallic and rare earth elements. This latter figure equates us with other locations where magnetic isotope data for Magnesite would make it worth pursuing such as Pennsylvania, New Jersey, San Nac/3 molar ratio studies, and a hexagonal ratio study, to name but a few. In this paper we provide perspectives in mineralogy and mineralogy variation of the assemblages of alternatives to Magnesite in West Virginia; as well as the communities that should feel comfortable championing the developers of M&RK and should consider the opportunity for they shops eventually reuse Magnesite in sites near their site, regardless of the comings and goings in North America. The mineralogy presented in this book is quite different from the one presented by Nides-R7K who has published articles and spoken series on Magnetozone suto or available water conditions provide t he following: i) They are complementary. ii) Reproducer: George Shee, CSU North West, University of Kansas. (Author book page#153-188), Valerie Schon, St. Paul Area Department of Science, University of Maine, University of Minnesota, Science Center, York University, Michigan State University, North Carolina State University, Ohio State University, University of Alabama-Birmingham, The College of chemist and analytical biologist, Society for Research on the Geology/Geology "Living with magnesites" 194 times since 1994: Number 737 How to cite this: Shee, George. Metals, minerals and past. Minerals, Minerals and Geology. Hostetter (D: Labour History of the Bureau of Mines), Washington, DC, 1996.
Here we report an efficient assay to differentiate, precisely and quickly, human prostate cancer (PCa) cell lines from normal cells/prostatic tissue. In prostatic PCa from each patient, we measured CE kinase activation of 1,2-bromopentaneidin (BPP), calcitonin-3, bone morphogenetic protein-6, urokinase 21 and eukaryotic phospho-protein kinase-2 (EPSK-2). Cells from 2 patients and 1 patient with hepatitis C and provided at the same time were simultaneously PCR screened to detect chromosomal chromosomal aberrations and restriction fragment length polymorphism variations (RFLP). We found one viable Gdp from each patient. We did not detect chromosomal aberrations and standard regulators of EGS in PCa patients. Most likely, our results, also for grade II PCa, show that PCa cells have a passive chromosomal integrity with DNA repair genes, but this is not found in intact androgen-depleted patients. Inadequate DNA repair genes and excessive chromosomal aberrations, due to non-genomic proportions of UGT-3, EDS1, EGS2, URP-88 and UAR6 compared to their value combined, lower DNA repair products of  $9.66\%$  , and reduced cell ploidy.(ABSTRACT TRUNCATED AT 250 WORDS)
During age-dependent contractions, the forearm flexor isomerophilic is a loss of maximum strength of the abductor tendon, which influences flexor mode and resistance. At peak contraction strength, the level of higher order strength produced by the concentric contractions is higher than the lower order strength due to lower upper, lower, and lateral potentiometric torque respectively. Moreover, age affects the time interval over which the elastic torque of the tendon increases, which changes from several short to several long and gradual. The apparent loss of maximal strain strength of the tendon during WER occurs generally late in the range of resistance, but in rare cases of progressively superior mechanical strength, the contractile effects of the WER series from the same period may be preserved.
Table 11: Examples of abstracts generated by GPT-2 fine-tuned on biomedical text.
Generated physics abstracts
We construct a free Kac-Moody algebra in terms of  $\mathrm{SU}(\mathbb{N})$  bi-holomorphic maps. Our construction is not exactly as in the standard program, which is based on the Kac-Moody algebra itself. In this work, we address the question whether a general construction can be made using a 3-dimensional space with its boundary fixed to be a manifold. For such a construction, one can quantize the free Kac-Moody algebra exactly using the interior data of a closed submanifold, an example of which is given.
A general classification of the massive scalar field model is presented. This model can be taken to be the simplest of all models exhibiting super-Kahler group structure. The very general model is then interpreted as a manifestation of the infinite supersymmetry of  $N = 2$  supersymmetric Yang-Mills theory with a simple Kaluza-Klein U(1) gauge group. This theory has an analog of Seiberg-Witten theory with a U(1) gauge group. The two models are related by the fact that the massive scalar field model reduces to the Seiberg-Witten model when the negative cosmological constant is eliminated.
We have simulated a sample of 60,000 galaxies in order to study the spatial structure of halos and the formation properties of their wind-driven columns. The galaxy models (average virial radius  $R_{c} = 0.25kpc$ , total column density  $O_{n}u^{-1 / 2} = 1.2cm^{-3}$ , total column density  $K_{0} = 0.3cm^{-3}$  and total column density  $L_{n}u - 2 / 3 = 0.5cm^{-3}$ ) were selected from a large database of spiral galaxy photometry. We have investigated the properties of the wind-driven column density as a function of the galaxy size, the radius and the column density in the inner Lyman limit.  $R_{c}$  and  $L_{n}u$  are defined as the fraction of the total column density,  $O_{n}u_{2} - L_{n}u$ , of gas in the central region of the galaxy, and  $L_{n}u$  is defined as the fractional logarithm of the gas density. The simulations were carried out on a large 64x64 grid of 16x16 square degrees (3.6 x 3.6 square degrees) in order to obtain a large number of potential galaxies. The cross-correlation of the radial profile of the radial intensity distribution and the velocity distributions of gas streams shows that the velocity distribution is strongly wave-like, with the velocity dispersion  $< 0.5km s^{-1}$ ; the power law of the dispersion is also found. We also find that the density of gas passes through a strong exponential regime at the center, and that the central density exceeds that at the periphery by 2 solar masses per degree of freedom.
We propose a new solution to the cosmic censorship problem. It is based on the idea that all physical experiments have to be canceled at the same time, with a specific choice of the data. In this approach the data would be separated from the source region by a detector, whose response is controlled by the Planck energy. The data would be split into beams of different energies. The different beams, and their response to each other, would be synchronized in a second detector, which would then (miraculously) detect photons from the source region and carry out the corresponding measurements in the detector. In this way the cosmic censorship problem can be reduced to a second problem: how to determine the signal and the background in the detector. We give an explicit example for the case of neutrino astronomy, and show that we can solve the cosmic censorship problem without any special choice of the signals of the detectors.