Using BibTeX to Automatically Generate Labeled Data for Citation Field Extraction

databases such as Google Scholar or Semantic Scholar . Citation field extraction ( CFE ) is precisely this task—given a reference label which tokens refer to the authors , venue , title , editor , journal , pages , etc . Most methods for CFE are supervised and rely on training from labeled datasets that are quite small compared to the great variety of reference formats . BIBTEX , the widely used reference management tool , provides a natural method to automatically generate and label training data for CFE . In this paper , we describe a technique for using BIBTEX to generate , automatically , a large-scale ( 41M labeled strings ) , labeled dataset , that is four orders of magnitude larger than the current largest CFE dataset , namely the UMass Citation Field Extraction dataset [ Anzaroot and McCallum , 2013 ] . We experimentally demonstrate how our dataset can be used to improve the performance of the UMass CFE using a RoBERTa-based [ Liu et al. , 2019 ] model . In comparison to previous SoTA , we achieve a 24.48 % relative error reduction , achieving span level F1-scores of 96.3 % . 1 . Introduction . Scholarly knowledge bases , such as Google Scholar and Semantic Scholar , are invaluable tools for navigating the landscape of scientific literature . Scholarly knowledge bases critically depend on the accurate bibliographic information to both populate research paper records ( containing information such as author , publication venue , publisher ) and provide citation graph information of which papers cite which other papers . While some such information is manually curated in sources such as PubMed , Web-of-Science , and others1 2 , the services are often incomplete ( missing tech reports , preprints , some books and conference proceedings ) or are not publicly available . Other services ( such as Google Scholar , Semantic Scholar , and CiteSeer ) aim to cover a wider set of academic papers by extracting relevant bibliographic information from a massive amount of unstructured sources collected from a variety of sources ( e.g. , web-crawls , conference proceedings ) . The quality of the information extraction tools impacts the quality of the database and in turn the user experience of the product and , as a result , the user ’ s research efficiency and effectiveness . 1. https : //www.crossref.org/ 2. https : //i4oc.org/ THAI , MONATH , XU , VEYTSMAN & MCCALLUM A core component of the automatic extraction of bibliographic information is citation field extraction , the task of segmenting a citation or reference string 3 into constituent parts ( fields ) , such as title , author ( s ) , publisher , and year . This also includes relatively uncommon fields such as language indicator , document type , and organization names . The size of the training data in the benchmark citation field extraction datasets is relatively small . The two mostly widely used labeled benchmarks , the most widely used labeled benchmarks , UMass Citation Field Extraction dataset [ Anzaroot and McCallum , 2013 ] contains 2476 references . However , references appear in a multitude of different formats throughout sciences and are often manually entered by scientists increasing the variability of their structure and formatting . Data augmentation [ Tran et al. , 2017 , Ratner et al. , 2017 ] and other automatic training data generation methods [ Tripathi et al. , 2019 ] have been shown to be highly effective in settings with limited training data . Examples of this include : closed captioned videos have been used to create automatically labeled speech recognition data [ Lakomkin et al. , 2018 ] , the automatic generation of 3D point cloud shapes has been used for self-driving vehicle systems [ Yue et al. , 2018 ] , and program synthesis to improve information extraction from political data and flight emails [ Iyer et al. , 2019 ] . Citation field extraction is uniquely well suited for these techniques using reference formatting tools , namely BIBTEX . BIBTEX can be used to format references in a wide variety of formatting styles and fonts , while using the structure of BIBTEX records to automatically label the reference fields for training and evaluating citation field extraction models . This provides us a mechanism to automatically generate labeled data . In this paper , we build a large-scale citation field extraction dataset which is automatically generated from publicly available BIBTEX files . We collect human-curated bibliographies from writing project repositories , authors and websites are collected and randomly paired with various styles to produce a set of reference strings . The references are aligned with their labels provided in the BIBTEX files to form labeling sequences . We collected 41M labeled references in total , twenty thousand times the size of the UMass CFE dataset . We train a variety of citation field extraction models including one based on RoBERTa [ Liu et al. , 2019 ] . We show that this model trained only on the UMass CFE dataset matches state-ofthe-art results [ Thai et al. , 2018 ] . We then show that training the BERT-based model on our large automatically generated dataset drastically improves the results , outperforming the state of the art approach by 1.2 points of F1 , a 24.48 % relative reduction in error . We then : show that certain subsets of our automatically generated dataset are considerably more challenging than existing benchmarks , present these datasets for evaluation in future work , and analyze our experimental results . All data and models for our approach are publicly available . 4 2 . Citation Field Extraction . Citation field extraction ( CFE ) is the task of segmenting a reference into its corresponding components ( Figure 1 ) . Given a reference , x , as a sequence of T tokens , x = { x1 , x2 , ... , xT } , CFE is the task of predicting the corresponding output sequence y = { y1 , y2 , ... , yT } where each output symbol yi is one of N possible bibliographic labels . 3 . Citation can refer to both in-line citations or the entries in the references section of a paper . As reference is less ambiguous we will use this despite the name of the task . 4 . URL withheld to preserve anonymity . USING BIBTEX TO AUTOMATICALLY GENERATE LABELED DATAFOR CITATION FIELD EXTRACTION We experiment with both structured prediction models that use a linear-chain conditional random field [ Lafferty et al. , 2001 ] for prediction as well as an independent prediction model using RoBERTabased representations . We summarized these models in this section . Structured Prediction Models The energy function for a particular configuration of the output sequence given the input is : E ( y|x ) = ∑T t=1 ψxy ( xt , yt ) + ψyy ( yt , yt+1 ) where the emission score or the local log-potentials ψxy is parameterized by a deep neural network , and the transition logpotentials ψyy are parameterized by an input-independent parameter matrix . Modeling the intra-state dependencies under a Markovian assumption , we get the data log-likelihood as : logP ( y|x ) = E ( y|x ) − log ∑ y′ exp ( E ( y′|x ) ) . Independent Predictions with RoBERTa Recent work on BERT and its variants [ Devlin et al. , 2019 , Liu et al. , 2019 ] has achieved state-of-the-art results on sequence tagging problems such as named entity recognition . We trained a RoBERTa model [ Liu et al. , 2019 ] to represent tokens in citation sequences . We then fine-tune this pretrained model for CFE task . For each token , the model makes an independent prediction of its class label . The prediction of the class label takes as input features of tokens from the last layer of the RoBERTa model and uses a linear classifier to predict the labels . 3 . Automatically Generating CFE Data . BIBTEX allows researchers to specify all fields of the reference as a structured record and render the reference in a specified style . Figure 1 shows an example record in its key-value format . BIBTEX provides a natural way to automatically generate labeled training data for CFE models . The labels of each field are defined by the key-value record , which in turn give labels to rendered strings . THAI , MONATH , XU , VEYTSMAN & MCCALLUM 3.1 Collecting BIBTEX . To create a large dataset of automatically labeled CFE examples , we collect BIBTEX records , curated by ( and hosted on the websites of ) researchers and organizations . Researchers tend to carefully curate their lists of publications due to the high value of these lists for job search and promotion . Collecting data from sources such as Google Scholar is problematic for several reasons , notably that manually curated records can not be distinguished from automatically extracted ones . We manually create a set of seed keywords from conference and journal listings websites , publishing venues and BIBTEX journal strings files . For example , some of our seeding words are “ Physics Letters ” , “ IEEE Transactions on ” , “ International Symposium on ” etc . Then , we construct a set of queries from these seeds and perform a web search to get BIBTEX files. ” We collected 6,023 BIBTEX files which contained 2,355,678 entries across various domains , a majority of entries being in Physics , Mathematics and Computer Science . Additionally , a smaller domain-specific dataset with more carefully chosen keywords was collected for analytical purposes . 3.2 Generating citation strings with labels . Different bibliography styles ( bst files ) can result in drastically different rendered strings . Figure 2 shows an example reference cited in different documents . To simulate this variation , we collect 269 BIBTEX styles from various sources such as CTAN and Overleaf . There are also a few options such as reference marker and hyphenation that can be modified from the TEX style rather than BIBTEX one . Due to the limitation of computing resources , we can not render every reference for all styles . We run minimal set coverage on the citation fields defined by these styles and select 26 styles that cover all citation fields . BIBTEX entries are randomly paired with BIBTEX styles and randomly chosen TEX options to produce citation strings in PDF format . As one can see from Figure 2 , citation fields in the resulting string can be omitted , abbreviated or rearranged depending on the style being used . Therefore , aligning the citation field and its label is a non-trivial task . To address this problem , we generate an additional marked reference string ( Figure 3 ) . In the marked string , the citation field values are marked up with their corresponding labels . Then , we simply use these markers to detect segment labels and boundaries . Sometimes field values themselves change during the generation process , such as abbreviating author names . In this case , we use the string distance algorithms to complete the alignment . USING BIBTEX TO AUTOMATICALLY GENERATE LABELED DATAFOR CITATION FIELD EXTRACTION Finally , we extract citation strings from PDF files . In the PDF file , a citation string may spread across multiple lines and these line segments may jump around when being converted to text due to PDF extracting errors . To minimize noises causing by the PDF extraction process , we produce a single PDF for each citation . We run dataset generation in parallel and generate roughly 41M labeled citation strings . Table 1 shows the summary statistics for our dataset . In Table 2 , we show the number of occurrences for some of the practical segment labels , both in the top high and low frequency . For the full segment counts of all 59 labels , see Appendix A . As shown in the tables , most of the labels of interest have a high number of supports . Label Number of segments author 91,324,094 year 52,946,966 title 42,846,934 journal 20,620,003 publisher 9,777,982 editor 3,481,227 institution 1,928,709 location 3,125 category 219 Table 2 : Segment counts for some labels of interest .

Summary: The paper is releasing a dataset and a model for "citation field extraction", the task of parsing the a bibliography entry of a paper into its corresponding fields (authors, title, venue, year ... etc). The dataset is automatically generated by collecting 6k bibtex files and 26 bibtex styles, then compile pdfs for the bibtex entries using multiple different styles. The resulting dataset is 41M labeled examples. The model starts with the RoBERTa checkpoint, continue MLM pretraining on this data, then finetune as a sequence labeler, which results into new sota performance. Error analysis shows that there's a big difference between the model performance on common fields (authors, title, .. ) and less common ones (edition, chapter).