SlowGuess/ACL_2020_OCR huggingface dataset:
two documents. Journal of the American Society for Information Science, 24(4):265-269.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Arnetminer: Extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, page 990-998, New York, NY, USA. Association for Computing Machinery.
Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103-110, Sydney, Australia. Association for Computational Linguistics.
Dominika Tkaczyk, Andrew Collins, Paraic Sheridan, and Joeran Beel. 2018. Machine learning vs. rules and out-of-the-box vs. retrained: An evaluation of open-source bibliographic reference and citation parsers. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, JCDL '18, page 99-108, New York, NY, USA. Association for Computing Machinery.
Caleb M. Trujillo and Tammy M. Long. 2018. Document co-citation analysis to enhance transdisciplinary research. Science Advances, 4(1).
Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Identifying meaningful citations. AAAI.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The Covid-19 Open Research Dataset.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv, abs/1609.08144.
Xiao Yu, Quanquan Gu, Mianwei Zhou, and Jiawei Han. 2012. Citation prediction in heterogeneous bibliographic networks. In SDM.
A Background & Terminology
In this work, we distinguish between bibliography entries and inline citations. A bibliography entry is an item in a paper's bibliography that refers to another paper. It is represented in a structured format that can be used for paper-identifying features such as title, authors, year, and venue or journal, and for journal articles, the volume, issue, and pages. Also commonly represented are unique document identifiers such as the Document Object Identifier (DOI), arXiv identifier, or PubMed identifier. Common formats for bibliography entries are MLA, APA, Vancouver-, and Chicago- style, among others, which are different ways of representing these various features for document identification.
There is often variation in the representation of certain fields. For example, Authors can include the first names of each author or only their first initials. In many academic disciplines, journal publications are the norm, whereas conference proceedings dominate in fields such as Computer Science; conference proceedings tend to lack journal-related features such as Volume, Issue, and Pages. Bibliography entry demarcation also varies between different formats. In some cases, each entry is preceded by a citation marker (e.g. "[1]" or "[ABC2019]") that is used throughout the text of the paper to denote inline citations.
An inline citation is a mention span within the paper's abstract or body text that refers to one of the entries in its bibliography.
"ABC (2019) present model 1, which outperforms model 2 (XYZ (2019)).
In this example, the narrative inline citation ABC (2019) appears as a noun phrase in the sentence while the parenthetical inline citation (XYZ, 2019) is inserted into the sentence as an aside. A sentence remains grammatically correct when parenthetical citations are removed. Other styles of parenthetical citations include, but are not limited to, BRACKET-style numbers (e.g. "[1, 3-5]") and OTHER styles such as superscripts (e.g. "1,2"), both of which refer to numbered entries in the bibliography. Bibliography entries without numbered entries or citation markers are typically referenced inline using NAME-YEAR format as ABC (2019) or (XYZ, 2019) in the example above.
Additionally, an inline reference is a span in a paper that refers to another part of the paper, for example, references to figures, tables, equations, proofs, sections, or appendices. These often take on the form of:
"In Figure 3, we show the relationship between A and B."
where Figure 3 refers to a plot displayed on a separate page. These inline references can be important for understanding the relationship between text and objects within the paper.
B PDF filters
Prior to running GROBID, we filter out PDFs that (i) produce an error when processed using the Python library PyPDF2,[20] (ii) have greater than 50 pages (more likely to be a dissertation or report), (iii) have page widths greater than page heights (more likely to be slides), and (iv) those which fail to be extracted using pdfalto, the variant of pdftoxml used by GROBID.
Numbers of PDFs removed by these filters are given in Table 8.
Filter	Number of PDFs
PyPDF2 error	0.54M
Over 50 pages	2.27M
Page width > height	0.28M
PDFAlto error	0.21M
Table 8: PDFs filtered out before GROBID processing
C The paper clustering problem
In academic fields in which preprint publishing is common (e.g. arXiv), the notion of a "paper" is somewhat ambiguous. For example, if a published paper differs from its arXiv preprint (as it often does), are the two documents considered separate papers for the purposes of citation? What about different arXiv preprint drafts tagged as different versions but under the same arXiv identifier?
In this work, each "paper" of interest is actually a collection (or cluster) of highly-similar (but not necessarily identical) documents. These paper clusters, provided by Semantic Scholar, are constructed to reflect how authors tend to view their
$^{20}$ Used to determine PDF page number and page dimensions
own papers; for example, most authors would consider their arXiv preprint and its associated published version to be the same "paper". For practical concerns in constructing S2ORC, we further require that one document within the cluster be the canonical document used to represent the paper cluster.
There are issues with defining a paper to be a collection of documents. For example, suppose a paper cluster contains both an arXiv preprint and a peer-reviewed draft. And suppose another paper cites the arXiv preprint critiquing content that has been updated in the peer-reviewed draft. If the peer-reviewed draft is chosen as the canonical representation of the paper cluster, then the citation context would not accurately capture the rationale of that reference. While worth noting, we believe such cases are rare and do not affect the vast majority of citation contexts.
D S2ORC evaluation criteria
Paper cluster quality For each paper cluster, we compare the selected canonical Title and Authors fields with the title and authors of the selected canonical PDF. The Title field is labeled correct if it exactly matches the title seen on the PDF, with some allowance for different capitalization and minor differences in special character representation (e.g. “ $\gamma$ ” versus “gamma”) and ignoring whitespace. The Authors field is labeled correct if all authors on the PDF are presented in the correct order, with some allowance for variation in the surface form. This is to avoid penalizing publisher metadata for providing a first initial (instead of the first name) or omitting middle names or titles (e.g. “Dr,” “PhD”).
Paper-Bibliography linking For each paper-bibliography pair, we compare the selected canonical Title and Authors fields in the structured bibliography entry to the selected canonical Title and Authors fields of the linked paper cluster. The Title fields are labeled as a match under the same criteria described above for matching paper cluster Title fields and PDF titles. The Authors fields are labeled as a match if there is substantial overlap in the names of the authors. For example, if authors A, B and C are in the bibliography entry and the linked paper cluster has authors A and B, then this is still considered a match. We note that in our evaluation, differences in the two sets of author names primarily stems from incorrectly writ
ten bibliography entries or mistakes in publisherprovided metadata.
E Training corpus sizes for other language models
Language model	Training data
ELMo(Peters et al., 2018a)	1BW (800M)Wikipedia (1.9B)WMT 2008-2012 (3.6B)
BERT(Devlin et al., 2019)	BooksCorpus (800M)Wikipedia (2.5B)
ROBERTA(Liu et al., 2019b)	BooksCorpus (800M)CC-News (~3.8B)OpenWebText (~1.9B)Stories (~1.6B)
GPT2(Radford et al., 2019)	Web Text Corpus (~2.8B)
Table 9: Reported and estimated (several papers report corpus size in terms of bytes) token counts of training data used to train language models.
We estimate that all of S2ORC consists of approximately 25B tokens of full body text and 15B tokens of abstract text. As demonstrated for S2ORC-SCiBERT pretraining, aggressively-cleaned body text from the PDF-parsed subset of S2ORC still yields approximately 16.5B tokens. The size of S2ORC makes it more than sufficient for pretraining large language models such as ELMo, BERT, RoBERTA, GPT2, and others, whose reported training data sizes are given in Table 9 for comparison.
Contextual Numeric Surface Forms, Layer 9
Figure 4: Visualization of contextual representations from layer 9 of S2ORC-SCIBERT on numeric surface forms in a subsample of body text from S2ORC. Labels are heuristics based on token-level patterns.
F Numeric representations in S2ORC-ScIBERT
Academic papers contain substantially more diverse uses of numeric surface forms than typical web text, such as experimental results, equations, citation references and section/figure markers. To demonstrate this, we cluster contextual word representations involving numbers, heuristically labeling them into one of 8 categories based on surface patterns. Examining the progression of the contextual representations through the layers of BERT reveals an initial focus on sentence position (expected, due to explicit position embeddings) and magnitude, with later layers integrating substantial contextual information, such as the presence of inline LATEX identifiers, citation indicators and PDF references. Following Peters et al. (2018b); Liu et al. (2019a), we observe that the final 2-3 BERT layers provide embeddings that excel at predictive language modeling; as such, Figure 4 uses embeddings from layer 9 of S2ORC-SCIBERT.