SlowGuess/NAACL_2021_OCR huggingface dataset:
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Computational Natural Language Learning (CoNLL), pages 147-155.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Computational Natural Language Learning (CoNLL), pages 142-147.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Association for Computational Linguistics (ACL), pages 2895-2905.
Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. 2019. Joint type inference on entities and relations via graph convolutional networks. In Association for Computational Linguistics (ACL), pages 1361-1370.
Bruno Taille, Vincent Guigue, Geoffrey Scoutheeten, and Patrick Gallinari. 2020. Let's stop incorrect comparisons in end-to-end relation extraction! In Empirical Methods in Natural Language Processing (EMNLP).
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Empirical Methods in Natural Language Processing (EMNLP), pages 5788-5793.
Jue Wang and Wei Lu. 2020. Two are better than one: Joint entity and relation extraction with table-sequence encoders. In Empirical Methods in Natural Language Processing (EMNLP).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 71-78.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2017a. End-to-end neural relation extraction with global optimization. In Empirical Methods in Natural Language Processing (EMNLP), pages 1730-1740.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017b. Position-aware attention and supervised data improve slot filling. In Empirical Methods in Natural Language Processing (EMNLP), pages 35-45.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Association for Computational Linguistics (ACL), pages 1441-1451.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In Association for Computational Linguistics (ACL), pages 1227-1236.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Association for Computational Linguistics (ACL), pages 427-434.
We use ACE04, ACE05, and SciERC datasets in our experiments. Table 2 shows the data statistics of each dataset.
The ACE04 and ACE05 datasets are collected from a variety of domains, such as newswire and online forums. We follow Luan et al. (2019)'s preprocessing steps $^{12}$  and split ACE04 into 5 folds and ACE05 into train, development, and test sets.
The SciERC dataset is collected from 12 AI conferences/workshop proceedings in four AI communities (Luan et al., 2018). SciERC includes annotations for scientific entities, their relations, and coreference clusters. We ignore the coreference annotations in our experiments. We use the processed dataset which is downloaded from the project website $^{13}$  of Luan et al. (2018).
We implement our models based on Hugging-Face's Transformers library (Wolf et al., 2019). For the entity model, we follow Wadden et al. (2019) and set the width embedding size as  $d_{F} = 150$  and use a 2-layer FFNN with 150 hidden units and ReLU activations to predict the probability distribution of entity types:
$$
P _ {e} (e \mid s _ {i}) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {e} \text {F F N N} \left(\mathbf {h} _ {e} \left(s _ {i}\right)\right)\right).
$$
For the relation model, we use a linear classifier on top of the span pair representation to predict the probability distribution of relation types:
$$
P _ {r} \left(r \mid s _ {i}, s _ {j}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {r} \mathbf {h} _ {r} \left(s _ {i}, s _ {j}\right)\right).
$$
For our approximation model (Section 4.3), we batch candidate pairs by adding 4 markers for each pair to the end of the sentence, until the total number of tokens exceeds 250. We train our models with Adam optimizer of a linear scheduler with a warmup ratio of 0.1. For all the experiments, we train the entity model for 100 epochs, and a learning rate of 1e-5 for weights in pre-trained LMs, 5e-4 for others and a batch size of 16. We train the relation model for 10 epochs with a learning rate of 2e-5 and a batch size of 32.
C Performance with Varying Data Sizes
We compare our pipeline model to a joint model with  $10\%$ ,  $25\%$ ,  $50\%$ ,  $100\%$  of training data on
Training data	Ours	Joint
Ent	Rel	Ent	Rel
10%	82.0	46.9	81.5	37.0
25%	84.9	57.6	84.6	49.0
50%	85.5	61.9	86.2	57.7
100%	87.2	63.4	87.4	61.0
Table 7: F1 scores on ACE05 development set when only a subset of training samples (10%, 25%, 50%, or 100%) are provided.
the ACE05 dataset. Here, our goal is to understand whether our finding still holds when the training data is smaller (and hence it is expected to have more errors in entity predictions).
Our baseline of joint model is our reimplementation of DYGIE++ (Wadden et al., 2019), without using propagation layers (the encoders are shared for the entity and relation model and no input marker is used; the top scoring  $0.4n$  entities are considered in beam pruning). As shown in Table 7, we find that our model achieves even larger gains in relation F1 over the joint model, when the number of training examples is reduced. This further highlights the importance of explicitly encoding entity boundaries and type features in data-scarce scenarios.
<sup>12</sup>We use the script provided by Luan et al. (2019): https://github.com/luanyi/DyGIE/tree/master/preprocessing.
13http://nlp.cs.washington.edu/sciIE/