SlowGuess/ACL_2020_OCR huggingface dataset:
Kyle Lo†* Lucy Lu Wang†* Mark Neumann† Rodney Kinney† Daniel S. Weld†‡
†Allen Institute for Artificial Intelligence
$^{\ddagger}$ Paul G. Allen School of Computer Science & Engineering, University of Washington
{kylel, lucyw}@allenai.org
We introduce S2ORC, a large corpus of 81.1M English-language academic papers spanning many academic disciplines. The corpus consists of rich metadata, paper abstracts, resolved bibliographic references, as well as structured full text for 8.1M open access papers. Full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. In S2ORC, we aggregate papers from hundreds of academic publishers and digital archives into a unified source, and create the largest publicly-available collection of machine-readable academic text to date. We hope this resource will facilitate research and development of tools and tasks for text mining over academic text.
Academic papers are an increasingly important textual domain for natural language processing (NLP) research. Aside from capturing valuable knowledge from humankind's collective research efforts, academic papers exhibit many interesting characteristics – thousands of words organized into sections, objects such as tables, figures and equations, frequent inline references to these objects, footnotes, other papers, and more.
Different types of resources have been used to support research over academic papers. Citation graphs like AMiner's Open Academic Graph (Tang et al., 2008), the Microsoft Academic Graph (MAG) (Shen et al., 2018), and the Semantic Scholar literature graph (Ammar et al., 2018), have had widespread application in bibliometrics, science-of-science, information retrieval, and network analysis. Digital archives like arXiv, $^{2}$
Figure 1: Inline citations and references to figures and tables are annotated in S2ORC's structured full text. Citations are linked to bibliography entries, which are linked to other papers in S2ORC. Figure and table references are linked to their captions.
PubMed Central, $^{3}$  CiteSeerX (Giles et al., 1998), $^{4}$  and the ACL Anthology (Bird et al., 2008), $^{5}$  are popular resources for deriving large text corpora for summarization and language modeling or, with further annotation, development of datasets for tasks like entity extraction, text classification, parsing, and discourse analysis. We focus on bibliometrically-enhanced derivations of these corpora, such as the ACL Anthology Network (AAN) (Radev et al., 2009) $^{6}$  derived from the ACL Anthology, RefSeer (Huang et al., 2015) derived from CiteSeerX, and Saier and Färber (2019) derived from arXiv, which combine useful aspects of citation graphs and raw text corpora. These resources provide citation mentions linked to paper identifiers in their corresponding digital archives, such as the ACL Anthology and CiteSeerX, or to nodes in citation graphs such as MAG, enabling new forms of cross-paper discourse analysis (e.g., studying how or why papers are related).
1 Instructions for access to the data and model are available at https://github.com/allenai/s2orc/.
$^{2}$ https://arxiv.org
<sup>3</sup>https://www.ncbi.nlm.nih.gov/PMC
<sup>4</sup>https://citeseerx.ist.psu.edu
<sup>5</sup>https://www.aclweb.org/anthology
http://aan.how/
4969
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983
Corpus	Papers w/ body text	Citation contexts	References to tables / figures / equations	Linked to graph	Academic disciplines
S2ORC (PDF-parse)	8.1M	full text	yes	S2ORC (full)	multi
S2ORC (LATEX-parse)	1.5M	full text	yes	S2ORC (full)	physics, math, CS
PubMed Central (OA)	2.6M	full text	yes	PubMed	bio, med
AAN (Radev et al., 2009)	25k	full text	no	ACL Anthology	comp ling
Saier and Färber (2019)†	1.0M	snippets	no	MAG	physics, math, CS
RefSeer (Huang et al., 2015)	1.0M	snippets	no	CiteSeerX	multi
Saier and Farber (2020) is an update to this work which now includes full text. It is released concurrently with this work.
Table 1: A comparison of S2ORC with other publicly-available academic text corpora. Of the other corpora: PubMed Central (OA) links to PubMed, which contains 30M papers at the time of writing. AAN links to the ACL Anthology (which contained 25k papers at the time of dataset construction, and 54k papers at the time of writing). Saier and Farber (2019) is derived from arXiv and links to MAG (which contained 213M papers and other non-paper documents at the time of dataset construction, and 226M nodes at the time of writing). RefSeer links to CiteSeerX (which contained 1M papers at the time of dataset construction, and 6M papers at the time of writing). S2ORC contains three times more full text papers than PubMed Central (OA), the next largest corpus with bibliometric enhancements, while covering a more diverse set of academic disciplines. Citations in S2ORC are linked to the full set of S2ORC papers, 81.1M paper nodes derived from Semantic Scholar. In addition, the LATEX subset of S2ORC captures additional structure omitted by Saier and Farber (2019), who also parse LATEX sources from arXiv.
Yet, existing corpora are not without their limitations. Some cover a small number of papers (e.g. AAN), are domain-specific (e.g. AAN, PubMed Central, Saier and Farber (2019)), or may not provide usable full text (e.g. Saier and Farber (2019) and RefSeer). To address these issues, we introduce S2ORC, the Semantic Scholar Open Research Corpus, a large publicly-available collection of 81.1M academic papers covering dozens of academic disciplines. Each paper is associated with metadata and abstracts aggregated from hundreds of trusted sources such as academic publishers and literature archives like PubMed and arXiv.
Notably, we release structured, machine-readable full text extracted from PDFs for 8.1M papers which we've identified as having open access status. S2ORC full text preserves meaningful structure, e.g., paragraph breaks, section headers, inline citation mentions, references to tables and figures, and resolved citation links to other papers. Additionally, we provide 1.5M full text LATEX parses from which we have extracted, in addition to citations and references, the source text of tables and mathematical formulas. As shown in Table 1, S2ORC provides substantially more structured full text papers and covers a more diverse set of academic disciplines than other resources.
In this paper, we describe the construction of S2ORC (§2). We provide summary statistics of the corpus (§3) and evaluate the data quality (§4). We then evaluate a BERT model pretrained on S2ORC (§5), and discuss potential applications to a variety of NLP and analysis tasks over academic text (§6). Finally, we compare S2ORC with other publicly-available academic text corpora (§7).
2 Constructing the corpus
S2ORC is constructed using data from the Semantic Scholar literature corpus (Ammar et al., 2018). Papers in Semantic Scholar are derived from numerous sources: obtained directly from publishers, from resources such as MAG, from various archives such as arXiv or PubMed, or crawled from the open Internet. Semantic Scholar clusters these papers based on title similarity and DOI overlap, resulting in an initial set of approximately 200M paper clusters.
To construct S2ORC, we must overcome challenges in (i) paper metadata aggregation, (ii) identifying open access publications, and (iii) clustering papers, in addition to identifying, extracting, and cleaning the full text and bibliometric annotations associated with each paper. The pipeline for creating S2ORC is:
1) Process PDFs and LATEX sources to derive metadata, clean full text, inline citations and references, and bibliography entries,
2) Select the best metadata and full text parses for each paper cluster,
7 pronounced "stork"
The papers included in S2ORC are a curated subset of the papers in the Semantic Scholar literature graph (Ammar et al., 2018) that focuses only on English-language papers with abstracts or full text available. See §2.5 for details on filtering through Semantic Scholar papers.
4970
3) Filter paper clusters with insufficient metadata or content, and
4) Resolve bibliography links between paper clusters in the corpus.
Details for these steps are provided below. See Appendix §A for definitions of terminology. The output of this pipeline is visualized in Figure 1.
2.1 Processing PDFs
We process PDFs from the Semantic Scholar corpus using SCIENCEparse v3.0 $^{9}$  and GROBID v0.5.5 $^{10}$  (Lopez, 2009). Our processing pipeline is described below.
Selecting PDFs We remove PDFs which are less likely to be academic papers. SCIENCEPARSE and GROBID are not optimized for processing non-paper academic documents such as dissertations, reports, slides, etc., and this filtering step is necessary to increase output data quality. See Appendix §B for filter details. There are around 31.3M PDFs associated with approximately 200M initial paper clusters, and 30.5M PDFs are selected for processing based on these filtering criteria.
Extracting structured data from PDFs We use SCIENCEparse to extract title and authors from each PDF.11 We then use GROBID to process each PDF. From the XML output of GROBID, we extract (i) metadata such as title, authors, and abstract, (ii) paragraphs from the body text organized under section headings, (iii) figure and table captions, (iv) equations, table content, headers, and footers, which we remove from the body text, (v) inline citations in the abstract and body text, (vi) parsed bibliography entries with title, authors, year, and venue identified, and (vi) links between inline citation mentions and their corresponding bibliography entries.
Postprocessing GROBID output We postprocess GROBID output using regular expressions to classify the parenthetical citation style of a paper as BRACKET (e.g. [2]), NAME-YEAR (e.g. ABC, 2019), or OTHER (superscripts and other mixed styles). We focus on addressing two types of common errors in GROBID's inline citation extractions: (i) false positives resulting from superscripts or equation references being recognized as
inline citations in papers with BRACKET-style citations, and (ii) false negatives resulting from an inability to expand bracket citation ranges (e.g. "[3]-[5]" should be expanded to "[3], [4], [5]" before linking). False positives are detected using regular expressions and removed from GROBID output. Bracket citation ranges are manually expanded and linked to their corresponding bibliography entries. The resulting parses are expressed in JSON format.[12]
2.2 Processing LATEX source
LATEX document source is available for a majority of arXiv submissions, and where available, are used to construct a full text parse. We retrieve body text, section headers, figure/table captions, table representations, equations, and inline citations and references directly from LATEX source. Inspired by Saier and Farber (2019), we first convert LATEX source into XML documents and then extract structured information from the XML.
Due to direct access to source, the accuracy of citation span, reference, caption, section header, and equation detection is near-perfect. We process 1.5M papers from LATEX source derived from arXiv, all of which are included as part of S2ORC. Surprisingly, due to the diversity of ways in which authors define metadata in LATEX, the quality of metadata extracted from LATEX documents is worse than those extracted from PDF. Therefore, we do not use LATEX-derived metadata for paper clustering or metadata selection.
2.3 Selecting canonical metadata
Canonical values for title, authors and other metadata fields are selected from among the papers in a cluster. First, if a cluster contains multiple PDFs, we select one to be canonical. This can occur, for example, in a cluster containing an arXiv preprint and its eventual camera-ready version. We preferentially select PDFs from open access sources and break ties by prioritizing PDFs for which there exist richer publisher-provided metadata (e.g. abstract, year, venue, DOI). If the selected PDF is associated with publisher-provided metadata, we select those publisher-provided metadata fields to be canonical.
In cases where publisher-provided metadata is incomplete, we use majority voting to select
12The S2ORC data format is described at https://github.com/allenai/s2orc
$^{9}$ https://github.com/allenai/science-parse
10https://github.com/kermitt2/grobid
11Our evaluations suggest SCIENCEPARSE outperforms GROBID for title and author extraction.
4971
canonical metadata values. We break ties by minimizing the total number of sources from which we select metadata (e.g., if IEEE provides title, authors and abstract, DBLP provides title and authors, and arXiv provides title and abstract, we prioritize selecting IEEE over the union of DBLP and arXiv). S2ORC metadata fields include title, author, year, venue, journal, abstract, and identifiers (DOI, PubMed, PubMed Central (PMC), arXiv, and ACL Anthology).
In cases where the title and authors are not provided by any publishers, we derive the values for these fields from the parsed PDF, prioritizing ScI-ENCEPARSE over GROBID. We further comment on paper clustering as it pertains to metadata selection in Appendix §C.
2.4 Assembling the corpus
We construct the final corpus by assembling clustered paper metadata with GROBID and LATEX parse objects. We associate the GROBID parse with the S2ORC paper object if a valid GROBID parse is produced from the PDF, and the PDF is open access. Open access status is assigned if a paper is derived from arXiv, ACL Anthology, PubMed Central (OA), and/or associated with an open-access DOI in the Unpaywall database.[13] If the PDF is not open access, we only include the bibliography from the GROBID parse in S2ORC. If arXiv LATEX source is available for the paper cluster, we also associate the LATEX parse with the S2ORC paper object.
2.5 Filtering paper clusters
We further filter paper clusters to remove papers with (i) no title, (ii) no authors, (iii) fewer than 100 characters of abstract and body text, and (iv) where English is not the primary language. The first three filters remove papers that provide little value for bibliometric-based or text-based analyses. The English language filter<sup>14</sup> reduces GRO-BID parsing errors. All filters are applied in series.
Subsequently, 95.5M paper clusters are filtered out based on the aforementioned criteria and removed from the corpus. The distribution of filtered papers is given in Table 2. We note that a large number of paper clusters are filtered out; 80.0M of these filtered clusters have no associated publisher-provided abstract or associated PDF and
do not provide significant value to our dataset in their current state. Although these papers that lack text may be useful as cite-able nodes in S2ORC, they are generally of lower quality and are filtered out of the corpus to improve corpus quality.
Filter	Number of papers
No title	20k
No authors	0.3M
< 100 chars of text	80.0M
Not English	15.2M
Table 2: Post-processing data quality filters for papers
2.6 Linking bibliographies to papers
Each bibliography entry in both GROBID and LA-TEX parses are linked to the most similar papers in the corpus. For linking, we score each bibliography entry and paper cluster pair using a similarity score computed between their titles. Each title is first normalized (i.e. white spaces stripped, lower-cased, special characters removed) and represented by its character 3-grams. The similarity score  $S_{title}$  is computed as the harmonic mean between a Jaccard index and a containment metric:
$$
S _ {t i t l e} = \frac {2 \times J \times C}{J + C} \tag {1}
$$
where the Jaccard index  $J$  and containment metric  $C$  are computed from the  $n$ -grams of the two titles  $N_{1}$  and  $N_{2}$  as:
$$
J = \frac {| N _ {1} \cap N _ {2} |}{| N _ {1} \cup N _ {2} |}
$$
$$
C = \frac {\left| N _ {1} \cap N _ {2} \right|}{\operatorname* {m i n} \left(\left| N _ {1} \right| , \left| N _ {2} \right|\right)}
$$
For each bibliography entry, the bibliography-paper pair with the highest similarity score above 0.8 is output as the correct link. Otherwise, the bibliography entry remains unlinked. We perform an evaluation of linking performance in §4.
3 The S2ORC dataset
The resulting corpus consists of 81.1M papers. Our publisher-provided abstract coverage is  $90.4\%$ , or 73.4M papers. Our PDF coverage is  $35.6\%$ , or 28.9M papers. These PDFs are processed using the pipeline discussed in §2.1. The
13Unpaywall 2019-04-19 data dump
14We use the cld2 tool for language detection with a threshold of 0.9 over the English language score.
Total papers	81.1M
Papers w/ PDF	28.9M (35.6%)
Papers w/ bibliographies	27.6M (34.1%)
Papers w/ GROBID full text	8.1M (10.0%)
Papers w/ LaTeX full text	1.5M (1.8%)
Papers w/ publisher abstract	73.4M (90.4%)
Papers w/ DOIs	52.2M (64.3%)
Papers w/ Pubmed IDs	21.5M (26.5%)
Papers w/ PMC IDs	4.7M (5.8%)
Papers w/ ArXiv IDs	1.7M (2.0%)
Papers w/ ACL IDs	42k (0.1%)
Table 3: Statistics on paper provenance. We note that categories are not mutually exclusive and do not sum to  $100\%$ . All papers in S2ORC have either a publisher-provided abstract or an associated PDF from which we derive full text and/or bibliography entries, or both.
Statistic	GROBID	LATEX
Paragraphs (abstract)	1.1	-
Paragraphs (body)	9.9	93.3*
Inline cite spans (abstract)	0.7	-
Inline cite spans (body)	45.2	46.8
Bibliography entries	27.6	21.9
Linked bib. entries	19.3	6.8†
*LATEX preserves line breaks rather than paragraph breaks.† The lower number of linked bibliography entries in LATEX parses is due to large numbers of papers (mostly in the field of physics) for which the bibliography entries are formatted without paper titles. Our linking algorithm strongly depends on titles and fails to link these entries.
Table 4: Extraction and linking statistics over PDF and LATEX parses. Reported values are averaged over all open access papers, which consist of 8.1M GROBID-parsed PDFs and 1.5M parsed LATEX sources.
VAST majority of these PDFs are successfully processed using GROBID, and we extract bibliography entries for 27.6M of the 28.9M PDFs. We identify 8.1M of the 28.9M PDFs as open access (§2.4), and we provide full text for all papers in this open access subset. For the 1.5M papers for which LATEX source is available through arXiv, we further obtain and provide LATEX parses (§2.2). Using these extracted bibliographies, we resolve a total 380.5M citation links between papers (§2.6), 156.5M of which can be tied back to their inline citation mentions in the full text. See Table 3 for more provenance statistics.
We provide statistics for the GROBID and LATEX full text parses and bibliography linking in
Distribution of papers by field of study
Figure 2: Distribution of papers by Microsoft Academic field of study.
Table 4. On average, LATEX parses contain many more "paragraphs" of body text, because LATEX source files preserve line breaks rather than paragraph breaks. We speculate that differences in bibliography entry and linking counts between the GROBID and LATEX parses are due to a combination of: (i) challenges in LATEX bibliography expansion and parsing, and (ii) differences in bibliography formatting in some math and physics venues (where bibliography entries do not include paper titles, which we depend on for bibliography linking).
The distribution of academic disciplines in S2ORC is given in Figure 2 using Microsoft Academic fields of study. Not all papers in S2ORC can be found in Microsoft Academic - those not found are denoted as Unclassified. Approximately  $677\mathrm{k}$  papers have more than one primary Microsoft Academic field of study; Figure 2 represents only the top field of study for each paper.
To evaluate the quality of our metadata selection, we randomly sample 500 paper clusters, restricting to those with PDFs. Within each sampled cluster, we determine whether the canonical title and authors match the title and authors in the selected canonical PDF.
Inline citation detection and bibliography parsing are dependent on GROBID (Lopez, 2009). Ahmad and Afzal (2018) evaluate GROBID for de
4973
Domain	Dataset	Reference	Task	SciBERT	S2ORC-SciBERT
Biomed	BC5CDR	Li et al. (2016)	NER	90.01	90.41 ± 0.06
JNLPBA	Collier and Kim (2004)	NER	77.28	77.70 ± 0.25
NCBI-disease	Doğan et al. (2014)	NER	88.57	88.70 ± 0.52
EBM-NLP	Nye et al. (2018)	PICO	72.28	72.35 ± 0.95
GENIA	Kim et al. (2003)	DEP (LAS)	90.43	90.80 ± 0.19
GENIA	Kim et al. (2003)	DEP (UAS)	91.99	92.31 ± 0.18
ChemProt	Krallinger et al. (2017)	REL	83.64	84.59 ± 0.93
CS	SciERC	Luan et al. (2018)	NER	67.57	68.93 ± 0.19
SciERC	Luan et al. (2018)	REL	79.97	81.77 ± 1.64
ACL-ARC	Jurgens et al. (2018)	CLS	70.98	68.45 ± 2.47
Biomed & CS	SciCite	Cohan et al. (2019)	CLS	85.49	84.76 ± 0.37
Multi-domain	PaperField	Beltagy et al. (2019)	CLS	65.71	65.99 ± 0.08
tecting inline citations using a corpus of 5k CiteSeer papers, and found GROBID to have an F1-score of 0.89 on this task. Tkaczyk et al. (2018) report GROBID as the best among 10 out-of-the-box tools for parsing bibliographies, also achieving an F1 of 0.89 in an evaluation corpus of 9.5k papers. We perform an evaluation over 200 randomly sampled papers from S2ORC and found comparable F1-scores for GROBID performance on both tasks.
For bibliography linking, we randomly sample S2ORC papers (500 GROBID PDF parses and 100 LATEX parses) and select one linked bibliography entry from each sampled paper (while avoiding selecting multiple entries linked to the same paper). We determine whether the title and authors in the bibliography entry agree with the title and authors of the linked paper.
We present these evaluation results in Table 6 and detail valuation criteria in Appendix §D.
Table 5: S2ORC-ScBERT test results are comparable with reported ScBERT test results on the set of tasks and datasets from Beltagy et al. (2019), to which we refer the reader for descriptions. Reported statistics are span-level F1 for NER, token-level F1 for PICO, dependency parsing (DEP), and macro-F1 for relation (REL) and text (CLS) classification. We report micro-F1 for ChemProt. All S2ORC-ScBERT results are the mean ± standard deviation of 5 runs with different random seeds. Beltagy et al. (2019) do not report standard deviation or number of runs.
Evaluated task	Title	Authors
Paper clustering	0.93	0.89
Bib. linking (GROBID)	1.00	0.96
Bib. linking (LATEX)	1.00	0.92
Table 6: Accuracy of paper clustering and bibliography linking for titles and authors in sampled evaluation sets.
5 Pretraining BERT on S2ORC
To demonstrate the suitability of S2ORC for language model pretraining, we train BERT-Base (Devlin et al., 2019) on the parsed full text of S2ORC and show that the resulting model (S2ORC-SCIBERT) performs similarly to SCIBERT (Beltagy et al., 2019) on a diverse suite of scientific NLP tasks and datasets.
While SCIBERT is a BERT-Base model also trained on multiple domains of scientific text, key differences in its pretraining corpus and vocabulary and those used for S2ORC-SCIBERT are:
- Domain: Beltagy et al. (2019) report a pretraining corpus consisting of  $82\%$  biomedical and  $18\%$  computer science papers. Our S2ORC pretraining corpus consists of a more balanced distribution of papers across diverse academic disciplines (see Figure 2), such that biomedical  $(42.7\%)$  and computer science  $(7.2\%)$  papers only comprise half the corpus.
- Preprocessing: S2ORC identifies figure captions, table text and captions, headers, footers, and footnotes. We exclude these from the pretraining corpus. We tokenize and sentencize the text using scispaCy (Neumann et al., 2019). We also use heuristic filters to remove ill-formed paragraphs (such as those containing too many symbols).
- Size: The resulting S2ORC pretraining cor
4974
pus contains 16.4B tokens, nearly five times larger than the corpus for ScIBERT.
- Vocab: Following Beltagy et al. (2019), we construct a cased WordPiece (Wu et al., 2016) vocabulary of size 31k using  $15\%$  of the S2ORC pretraining corpus. The Jaccard index between the S2ORC-SCIBERT and SCIBERT vocabularies is 0.536.