SlowGuess/ACL_2020_OCR huggingface dataset:
For each cluster  $C_i \in R$ , we construct its section embedding  $E_i^s$  by max-pooling span embeddings of the mentions of  $C_i$  that occur in section  $s$  (along with a learned bias vector  $b$  in case no mentions of  $C_i$  appear in section  $s$ ). Then the section  $s$  embedding of tuple  $R$  is  $E_R^s = \mathrm{FFN}([E_1^s; E_2^s; E_3^s; E_4^s])$  where ; denotes concatenation and FFN is a feedforward network. We then construct a document level embedding of  $R$ ,  $E_R$  as mean of section embeddings  $\frac{1}{|S|} \sum_{s=1}^{|S|} E_R^s$ . The final classification for relationship is done by passing the  $E_R$  through another FFN, which returns a probability of this tuple expressing a relation in this document.
Training Procedure While mention identification, span saliency classification, and relation extraction share the base document and span representation from BERT + BiLSTM and trained jointly, each of these subparts is trained on ground truth input. Note that we require the saliency classification and relation extraction to be independent of mention identification task since the output of this task (essentially the span of mention text) is non
7511
differentiable. The model jointly optimizes three losses, negative log-likelihood for mention identification, binary cross-entropy for saliency classification, and binary cross-entropy for relation extraction, with all three losses weighted equally.
We compare our model with other recently introduced models. Since we cannot apply previous models directly to our task, we evaluate on subtasks of our dataset and also evaluate on ScIERC (Section 5.2). The other goal of the evaluation is to establish a baseline performance on our dataset and to provide insights into the difficulty of each subtask. To that end, we evaluate the performance of each component separately (Section 5.3), and in the overall end-to-end system (Section 5.4). In addition, we perform diagnostic experiments to identify the bottlenecks in the model performance. We report experimental setup and hyperparameters in appendix A.
Mention Identification is a sequence labeling task, which we evaluate using the standard macro average F1 score of exact matches of all mention types.
Salient Mentions and Pairwise Coreference are binary classification tasks which we evaluate using the F1 score.
Salient Entity Clustering evaluation relies on some mapping between the set of predicted clusters and gold clusters. Given a predicted cluster  $\mathcal{P}$  and a gold cluster  $\mathcal{G}$ , we consider  $\mathcal{P}$  to match  $\mathcal{G}$  if more than  $50\%$  of  $\mathcal{P}$ 's mentions belong to  $\mathcal{G}$ , that is  $\frac{|\mathcal{P} \cap \mathcal{G}|}{|\mathcal{P}|} > 0.5$ . The 0.5 threshold enjoys the property that, assuming all predicted clusters are disjoint from each other (which is the case by construction) and gold clusters are disjoint from each other (which is the case for  $98.5\%$  of them), a single predicted cluster can be assigned to at most one gold cluster. This maps the set of predicted clusters to gold clusters, and given the mapping, it is straightforward to use the F1 score to evaluate predictions. This procedure optimizes for identifying all gold clusters even if they are broken into multiple predicted clusters.
Relation Extraction evaluation relies on the same mapping used in the evaluation of salient entity clustering. Under such mapping, each predicted  $N$ -ary relation can be compared with gold relations, and decide if they match or not. This becomes a binary classification task that we evaluate with positive class F1 score. We report F1 scores for binary and 4-ary relation tuples. We get binary relations by splitting each 4-ary relation into six binary ones.
5.2 Comparing with Baselines
We compare our model with DYGIE++ (Wadden et al., 2019) and DocTAET (Hou et al., 2019) on subtasks of our SCIREX dataset and on the SCIERC dataset wherever they apply. Our results show that only our model can perform all the subtasks in an end-to-end fashion and performs better than or on par with these baselines on respective subtasks.
5.2.1 Evaluation on SCIREX
$\mathsf{D}\mathsf{Y}\mathsf{G}\mathsf{E} + +$  (Wadden et al., 2019) is an end-to-end model for entity and binary relation extraction (check Section 2 for details). Being a span enumeration type model,  $\mathsf{D}\mathsf{Y}\mathsf{G}\mathsf{E} + +$  only works on paragraph level texts and extracts relations between mentions in the same sentence only. Therefore, we subdivide SCIREX documents into sections and formulate each section as a single training example. We assume all entities in relations returned by  $\mathsf{D}\mathsf{Y}\mathsf{G}\mathsf{E} + +$  are salient. We map each binary mention-level relation returned to entity-level by mapping the span to its gold cluster label if it appears in one. We consider 3 training configurations of  $\mathsf{D}\mathsf{Y}\mathsf{G}\mathsf{E} + +$ , 1. trained only on the abstracts in our dataset, 2. trained on all sections of the documents in our dataset. 3. trained on SCIERC dataset (still evaluated on our dataset), At test time, we evaluate the model on all sections of the documents in the test set.
Results in Table 3 show that we perform generally better than DYGIE++. The performance on end-to-end binary relations shows the utility of incorporating a document level model for cross-section relations, rather than predicting on individual sections. Specifically, We observe a large difference in recall, which agrees with the fact that  $55\%$  of binary relation occur across sentence level. DYGIE++ (All sections) were not able to identify any binary relations because  $80\%$  of training examples have no sentence level binary relations, pushing the model towards predicting very few relations. In
It is conceivable that mixing the gold mention spans with predicted mention spans might give an improvement in performance; therefore, we leave this as future work.
<sup>7</sup>We consider two mention spans to be a match if their Jaccard similarity is greater than 0.5.
Model	P	R	F1
Mention Identification
DYGIE++	0.703	0.676	0.678
Our Model	0.707	0.717	0.712
End-to-end binary relations
DYGIE++ (Abstracts Only)	0.003	0.001	0.002
DYGIE++ (All sections)	0.000	0.000	0.000
DYGIE++ (SciERC)	0.029	0.128	0.038
Our Model	0.065	0.411	0.096
4-ary relation extraction only
DocTAET	0.477	0.885	0.619
Our Model	0.531	0.718	0.611
Table 3: Evaluating state-of-the-art models on subtasks of ScIREX dataset because we did not find an existing model that can perform the end-to-end task.
Task	Model	P	R	F1
Mention Ident.	DYGIE++	0.676	0.694	0.685
Our Model	0.637	0.640	0.638
Pairwise Coref. and Clustering	DYGIE++	0.577	0.455	0.476
Our Model	0.187	0.552	0.255
contrast, training on SCIERC (and evaluating on SCIREX) gives better results because it is still able to find the few sentence-level relations.
DocTAET (Hou et al., 2019) is a document-level relation classification model that is given a document and a relation tuple to classify if it is expressed in the document. It is formulated as an entailment task with the information encoded as [CLS] document [SEP] relation in a BERT style model. This is equivalent to the last step of our model but with gold salient entity clusters as input. Table 3 shows the result on this subtask, and it shows that our relation model gives comparable performance (in terms of positive class F1 score) to that of DocTAET.
5.2.2 Evaluation on ScIERC
Table 4 summarizes the results of evaluating our model and DYGIE++ on the ScIERC dataset. For mention identification, our model performance is a bit worse mostly because ScIERC has overlapping entities that a CRF-based model like ours can not handle. For the task of identifying coreference clusters, we perform significantly worse than DYGIE++'s end-to-end model. This provides future avenues towards improving coreference resolution for ScIREX by incorporating it in an end-to-end fashion.
Table 4: Comparison of DYGIE++ with our model on various subtasks of SCIERC dataset
Task	P	R	F1
Component-wise (gold Input)
Mention Identification	0.707	0.717	0.712
Pairwise Coreference	0.861	0.852	0.856
Salient Mentions	0.575	0.584	0.579
Salient Entity Clusters	1.000	0.984	0.987
Binary Relations	0.820	0.440	0.570
4-ary Relations	0.531	0.718	0.611
End-to-end (predicted input)
Salient Entity Clusters	0.223	0.600	0.307
Binary Relations	0.065	0.411	0.096
4-ary Relations	0.007	0.173	0.008
End-to-end (gold salient clustering)
Salient Entity Clusters	0.776	0.614	0.668
Binary Relations	0.372	0.328	0.334
4-ary Relations	0.310	0.281	0.268
Table 5: Analysis of performance of our model and its subtasks under different evaluation configurations.
5.3 Component-wise Evaluation
The main contribution of our model is to connect multiple components to perform our end-to-end task. This section evaluates each step of our model separately from all other components. To do so, we feed each component with gold inputs and evaluate the output. This gives us a good picture of the performance of each component without the accumulation of errors.
The first block of Table 5 summarizes the results of this evaluation setting. We know from Tables 3, 4 that our mention identification and relation identification components are working well. For pairwise coreference resolution, we know from Table 4 that it needs to be improved, but it is performing well on our dataset likely because the majority of coreferences in our dataset can be performed using only the surface form of the mentions (for example, abbreviation reference). The worst performing component is identifying salient mentions, which requires information to be aggregated from across the document, something the current neural models lack.[8]
5.4 End-to-End Evaluation
Evaluation with Predicted Input. The second block in Table 5 gives results for the end-to-end performance of our model in predicting salient entity clusters, binary relations, and 4-ary relations. We noticed that there is quite a drop in the end-to
<sup>8</sup>Performance of Salient Entity Clusters is close to 1.0 because it is a deterministic algorithm (clustering followed by filtering) that gives perfect output given gold input. The reason the recall is not 1.0 as well is because of small inconsistencies in the gold annotations (two distinct entities merged into one).
end performance compared to the component-wise performance. This is particularly clear with relations; even though the relation extraction component performance is reasonably good in isolation, its end-to-end performance is quite low because of the accumulation of errors in previous steps.
Evaluation with Gold Salient Clustering.
Through manual error analysis, we found that the identification of salient clusters is the most problematic step in our model. The third block in Table 5 quantifies this. In this setting, we run our end-to-end model but with "gold cluster saliency" information. In particular, we predict clusters of mentions using our model (mention identification, pairwise coreference, and mention clustering). Then instead of filtering clusters using our mention saliency score, we keep only those clusters that have any overlap with at least one gold cluster. Predicted clusters that match the same gold cluster are then combined. Finally, we feed those to the relation extraction step of our model. Under this setting, we found that the performance of 4-ary relations improves considerably by more than  $10\mathrm{x}$ . This confirms our hypothesis that identifying salient clusters is the key bottleneck in the end-to-end system performance. This is also consistent with the component-wise results that show low performance for salient mentions identification.
Error Analysis for Identifying Salient Clusters.
Our error analysis shows that the average number of mentions in a salient cluster classified correctly is 15 mentions, whereas for the misclassified ones is six mentions. This indicates that our model judges the saliency of an entity strongly based on how frequently it is mentioned in the document. While this is a perfectly reasonable signal to rely on, the model seems to trust it more than the context of the entity mention. For example, in the following snippet, "... For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropProb, and the computational efficiency ....", the entity "the parameter counts" is misclassified as non-salient, as it only appears twice in the document. One possible way to address this issue with salient entity identification is to replace its simple filtering step with a trained model that can do a better job at aggregating evidence from multiple mentions.
Overall, these results indicate that identifying the saliency of entities in a scientific document is
a challenging task. It requires careful document-level analysis, and getting it right is crucial for the performance of an end-to-end document-level IE model. Also, the difference between results in the third block of the results and the component-wise results indicate that the whole model can benefit from incremental improvements to each component.
We introduce SCIREX, a comprehensive and challenging dataset for information extraction on full documents. We also develop a baseline model for our dataset, which, to the best of our knowledge, is the first attempt toward a neural document level IE that can perform all the necessary subtasks in an end-to-end manner. We show that using a document level model gave a significant improvement in terms of recall, compared to existing paragraph-level approaches.
This task poses multiple technical and modeling challenges, including 1. the use of transformer-based models on long documents and related device memory issues, 2. aggregating coreference information from across documents in an end-to-end manner, 3. identifying salient entities in a document and 4. performing N-ary relation extraction of these entities. Each of these tasks challenges existing methodologies in the information extraction domain, which, by and large, focus on short text sequences. An analysis of the performance of our model emphasizes the need for better document-level models that can overcome the new challenges posed by our dataset. As our research community moves towards document level IE and discourse modeling, we position this dataset as a testing ground to focus on this important and challenging task.
This research was supported by the ONR MURI N00014-18-1-2670, ONR N00014-18-1-2826, DARPA N66001-19-2-4031, and Allen Distinguished Investigator Award. We thank the Semantic Scholar team at AI2, UW NLP, and anonymous reviewers for their insightful comments. We are especially grateful to Kyle Lo for help with Grobid parser, the complete Papers With Code team for making their data publicly available, Dan Weld and Robert Stojnic for helpful discussion and feedback.
7514
2008-2020. Grobid. https://github.com/ kermitt2/grobid.
Gabor Angeli, Victor Zhong, Danqi Chen, Arun Tejasvi Chaganty, Jason Bolton, Melvin Jose Johnson Premkumar, Panupong Pasupat, Sonal Gupta, and Christopher D. Manning. 2015. Bootstrapped self training for knowledge base population. Theory and Applications of Categories.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2017. Overview of linguistic resources for the tac kbp 2017 evaluations: Methodologies and results. Theory and Applications of Categories.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
Jeremy Getman, Joe Ellis, Stephanie Strassel, Zhiyi Song, and Jennifer Tracey. 2018. Laying the groundwork for knowledge base population: Nine years of linguistic resources for tac kbp. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level n-ary relation extraction with multiscale representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Zhanming Jie and Wei Lu. 2019. Dependency-guided LSTM-CRF for named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China. Association for Computational Linguistics.
David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi-supervised neural tagging. In Proceedings of Empirical Methods in Natural Language Processing.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of The North American Chapter of the Association for Computational Linguistics (NAACL).
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. Association for Computational Linguistics.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics.
Rashedur Rahman, Brigitte Grau, Sophie Rosset, Yoann Dupont, Jérémy Guillemot, Olivier Mesnard, Christian Lautier, and Wilson Fred. 2016. Tac kbp 2016 cold start slot filling and slot filler validation systems by irt systemx. Theory and Applications of Categories.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer.
Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
Chen-Tse Tsai, Gourab Kundu, and Dan Roth. 2013. Concept-based analysis of scientific literature. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
Joe H. Ward. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301).
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
We divide our 438 annotated documents into training (70%), validation (30%) and test set (30%). The base document representation of our model is formed by SciBERT-base (Beltagy et al., 2019) and BiLSTM with 128-d hidden state. We use a dropout of 0.2 after BiLSTM embeddings. All feedforward networks are composed of two hidden layers, each of dimension 128 with gelu activation and with a dropout of 0.2 between layers. For additive attention layer in span representation, we collapse the token embeddings to scalars by passing through
the feedforward layer with 128-d hidden state and performing a softmax. We train our model for 30 epochs using Adam optimizer with 1e-3 as learning rate for all non BERT weights and 2e-5 for BERT weights. We use early stopping with a patience value of 7 on the validation set using relation extraction F1 score. All our models were trained using 48Gb Quadro RTX 8000 GPUs. The multitask model takes approximately 3 hrs to train.
For the BERT coreference model, we use SciBERT-base embeddings with two mentions encoded as [CLS] mention 1 [SEP] mention 2 [SEP]. We use a linear layer on top of [CLS] token embedding to compute the mention pair's coreference score.
All our models were implemented in AllenNLP library(Gardner et al., 2017).
B Annotation Guidelines
Our Annotation guidelines can be found at https://github.com/allenai/SciREX/blob/master/Annotation%20Guidelines.pdf Note, for Method type entities, we specifically ask our annotator to break down complex entities into simpler ones before looking for mentions in the text. For example, a method entity DLDL+VGG-Face is composite and broken into two parts DLDL and VGG-Face. Currently, our model considers all mentions of subentities as mentions of the corresponding Method entity. We leave the task of extracting relation between subentities explicitly as future work.
7516