SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	Encoder	ACE05	ACE04	SciERC
Ent	Rel	Rel+	Ent	Rel	Rel+	Ent	Rel	Rel+
(Li and Ji, 2014)	-	80.8	52.1	49.5	79.7	48.3	45.3	-	-	-
(Miwa and Bansal, 2016)	L	83.4	-	55.6	81.8	-	48.4	-	-	-
(Katiyar and Cardie, 2017)	L	82.6	55.9	53.6	79.6	49.3	45.7	-	-	-
(Zhang et al., 2017a)	L	83.6	-	57.5	-	-	-	-	-	-
(Luan et al., 2018)♣†	L+E	-	-	-	-	-	-	64.2	39.3	-
(Luan et al., 2019)♣†	L+E	88.4	63.2	-	87.4	59.7	-	65.2	41.6	-
(Li et al., 2019)	Bl	84.8	-	60.2	83.6	-	49.4	-	-	-
(Dixit and Al-Onaizan, 2019)	L+E	86.0	-	62.8	-	-	-	-	-	-
(Wadden et al., 2019)♣†	Bb	88.6	63.4	-	-	-	-	-	-
(Wadden et al., 2019)♣†	SciB	-	-	-	-	-	-	67.5	48.4
(Lin et al., 2020)	Bl	88.8	67.5	-	-	-	-	-	-	-
(Wang and Lu, 2020)	ALB	89.5	67.6	64.3	88.6	63.3	59.6	-	-	-
PURE (ours): single-sentence	Bb	88.7	66.7	63.9	88.1	62.8	58.3	-	-	-
SciB	-	-	-	-	-	-	66.6	48.2	35.6
ALB	89.7	69.0	65.6	88.8	64.7	60.2	-	-	-
PURE (ours): cross-sentence♣	Bb	90.1	67.7	64.8	89.2	63.9	60.1	-	-	-
SciB	-	-	-	-	-	-	68.9	50.1	36.8
ALB	90.9	69.4	67.0	90.3	66.1	62.2	-	-	-
Table 1: Test F1 scores on ACE04, ACE05, and SciERC. We evaluate our approach in two settings: single-sentence and cross-sentence depending on whether cross-sentence context is used or not.  $\clubsuit$ : These models leverage cross-sentence information.  $\dagger$ : These models are trained with additional data (e.g., coreference). The encoders used in different models: L = LSTM, L+E = LSTM + ELMo, Bb = BERT-base, Bl = BERT-large, SciB = SciBERT (size as BERT-base), ALB = ALBERT-xxlarge-v1. Rel denotes the boundaries evaluation (the entity boundaries must be correct) and  $Rel+$  denotes the strict evaluation (both the entity boundaries and types must be correct).
Dataset	|E|	|R|	# Sentences
Train	Dev	Test
ACE05	7	6	10,051	2,424	2,050
ACE04	7	6	8,683 (5-fold)
SciERC	6	7	1,861	275	551
Table 2: The statistics of the datasets. We use ACE04, ACE05, and SciERC for evaluating end-to-end relation extraction.
the relation model in our default setting using cross-sentence context $^{8}$  and the effect of different context sizes is provided in Section 5.4. We consider spans up to  $L = 8$  words. For all the experiments, we report the averaged F1 scores of 5 runs. More implementation details can be found in Appendix B.
Table 1 compares our approach PURE to all the previous results. We report the F1 scores in both single-sentence and cross-sentence settings. As is shown, our single-sentence models achieve strong performance and incorporating cross-sentence con
text further improves the results considerably. Our BERT-base (or SciBERT) models achieve similar or better results compared to all the previous work including models built on top of larger pre-trained LMs, and our results are further improved by using a larger encoder ALBERT.
For entity recognition, our best model achieves an absolute F1 improvement of  $+1.4\%$ ,  $+1.7\%$ ,  $+1.4\%$  on ACE05, ACE04, and SciERC respectively. This shows that cross-sentence information is useful for the entity model and pre-trained Transformer encoders are able to capture long-range dependencies from a large context. For relation extraction, our approach outperforms the best previous methods by an absolute F1 of  $+1.8\%$ ,  $+2.8\%$ ,  $+1.7\%$  on ACE05, ACE04, and SciERC respectively. We also obtained a  $4.3\%$  higher relation F1 on ACE05 compared to DYGIE++ (Wadden et al., 2019) using the same BERT-base pre-trained model. Compared to the previous best approaches using either global features (Lin et al., 2020) or complex neural models (e.g., MT-RNNs) (Wang and Lu, 2020), our approach is much simpler and achieves large improvements on all the datasets. Such improvements demonstrate the effectiveness
<sup>8</sup>We use a context window size  $W = 100$  for the ALBERT entity models to reduce GPU memory usage.
Model	ACE05	SciERC
Rel (F1)	Speed (sent/s)	Rel (F1)	Speed (sent/s)
Full (single)	66.7	32.1	48.2	34.6
Approx. (single)	65.7	384.7	47.0	301.1
Full (cross)	67.7	14.7	50.1	19.9
Approx. (cross)	66.5	237.6	48.8	194.7
Table 3: We compare our full relation model and the approximation model in both accuracy and speed. The accuracy is measured as the relation F1 (boundaries) on the test set. These results are obtained using BERT-base for ACE05 and SciBERT for SciERC in both single-sentence and cross-sentence settings. The speed is measured on a single NVIDIA GeForce 2080 Ti GPU with a batch size of 32.
of learning representations for entities and relations of different entity pairs, as well as early fusion of entity information in the relation model. We also noticed that compared to the previous state-of-the-art model (Wang and Lu, 2020) based on ALBERT, our model achieves a similar entity F1 (89.5 vs 89.7) but a substantially better relation F1 (67.6 vs 69.0) without using context. This clearly demonstrates the superiority of our relation model. Finally, we also compare our model to a joint model (similar to DYGIE++) of different data sizes to test the generality of our results. As shown in Appendix C, our findings are robust to data sizes.
4.3 Batch Computations and Speedup
In Section 3.3, we proposed an efficient approximation solution for the relation model, which enables us to re-use the computations of text tokens and batch multiple span pairs in one input sentence. We evaluate this approximation model on ACE05 and SciERC. Table 3 shows the relation F1 scores and the inference speed of the full relation model and the approximation model. On both datasets, our approximation model significantly improves the efficiency of the inference process. For example, we obtain a  $11.9 \times$  speedup on ACE05 and a  $8.7 \times$  speedup on SciERC in the single-sentence setting. By re-using a large part of computations, we are able to make predictions on the full ACE05 test set (2k sentences) in less than 10 seconds on
a single GPU. On the other hand, this approximation only leads to a small performance drop and the relation F1 measure decreases by only  $1.0\%$  and  $1.2\%$  on ACE05 and SciERC respectively in the single-sentence setting. Considering the accuracy and efficiency of this approximation model, we expect it to be very effective to use in practice.
Despite its simple design and training paradigm, we have shown that our approach outperforms all previous joint models. In this section, we aim to take a deeper look and understand what contributes to its final performance.
5.1 Importance ofTyped Text Markers
Our key observation is that it is crucial to build different contextual representations for different pairs of spans and an early fusion of entity type information can further improve performance. To validate this, we experiment the following variants on both ACE05 and SciERC:
TEXT: We use the span representations defined in the entity model (Section 3.2) and concatenate the hidden representations for the subject and the object, as well as their element-wise multiplication:  $\left[\mathbf{h}_e(s_i),\mathbf{h}_e(s_j),\mathbf{h}_e(s_i)\odot \mathbf{h}_e(s_j)\right]$ . This is similar to the relation model in Luan et al. (2018, 2019).
TEXTETYPE: We concatenate the span-pair representations from TEXT with entity type embeddings  $\psi(e_i), \psi(e_j) \in \mathbb{R}^{d_E}$  ( $d_E = 150$ ).
MARKERS: We use untyped entity types  $(\langle S\rangle, \langle /S\rangle, \langle O\rangle, \langle /O\rangle)$  at the input layer and concatenate the representations of two spans's starting points.
MARKERS: We concatenate the span-pair representations from MARKERS with entity type embeddings  $\psi(e_i), \psi(e_j) \in \mathbb{R}^{d_E}$  ( $d_E = 150$ ).
MARKERSELOSS: We also consider a variant which uses untyped markers but add another FFNN to predict the entity types of subject and object through auxiliary losses. This is similar to how the entity information is used in multi-task learning (Luan et al., 2019; Wadden et al., 2019).
TypedMARKERS: This is our final model described in Section 3.2 with typed entity markers.
Table 4 summarizes the results of all the variants using either gold entities or predicted entities from the entity model. As is shown, different input representations make a clear difference and the variants of using marker tokens are significantly
Note that we only applied this batch computation trick at inference time, because we observed that training with batch computation leads to a slightly (and consistently) worse result. We hypothesize that this is due to the impact of increased batch sizes. We still modified the position embedding and attention masks during training (without batching the instances though).
Input	ACE05	SciERC
gold	e2e	gold	e2e
TEXT	67.6	61.6	61.7	45.3
TEXTETYPE	68.2	62.6	63.6	45.7
MARKERS	70.5	63.3	68.2	49.1
MARKERSETYPE	71.3	63.8	68.9	49.7
MARKERSELOSS	70.7	63.6	68.0	49.2
TYPEDMARKERS	72.6	64.2	69.1	49.7
Table 4: Relation F1 (boundaries) on the development set of ACE05 and SciERC with different input features.  $e2e$ : the entities are predicted by our entity model; gold: the gold entities are given. The results are obtained using BERT-base with single-sentence context for ACE05 and SciBERT with cross-sentence context for SciERC. For both ACE05 and SciERC, we use the same entity models with cross-sentence context to compute the  $e2e$  scores of using different input features.
Shared encoder?	Entity F1	Relation F1
×	88.8	64.8
✓	87.7	64.4
better than standard text representations and this suggests the importance of learning different representations with respect to different pairs of spans. Compared to TEXT, TYPEDMARKERS improved the F1 scores dramatically by  $+5.0\%$  and  $+7.4\%$  absolute when gold entities are given. With the predicted entities, the improvement is reduced as expected while it remains large enough. Finally, entity type is useful in improving the relation performance and an early fusion of entity information is particularly effective (TYPEDMARKERS vs MARKERSETYPE and MARKERSELOSS). We also find that MARKERSETYPE to perform even better than MARKERSELOSS which suggests that using entity types directly as features is better than using them to provide training signals through auxiliary losses.
5.2 Modeling Entity-Relation Interactions
One main argument for joint models is that modeling the interactions between the two tasks can contribute to each other. In this section, we aim to validate if it is the case in our approach. We first study whether sharing the two representation encoders can improve performance or not. We train the entity and relation models together by jointly
Table 5: Relation F1 (boundaries) scores when entity and relation encoders are shared and not shared on the ACE05 development set. This result is obtained from BERT-base models with cross-sentence context.
ACE05	SciERC
Gold entities	64.8	49.7
10-way jackknifing	63.9	48.1
0.4n spans (typed)	64.6	50.2
0.4n spans (untyped)	56.9	48.4
0.4n spans (untyped + eloss)	63.0	48.5
Table 6: We compare relation F1 (boundaries) with different training strategies on the development sets of ACE05 and SciERC. This result is from training BERT-base and SciBERT models with cross-sentence context. typed: typed markers, untyped: untyped markers, untyped + eloss: untyped markers with auxiliary entity loss. See text for more details.
optimizing  $\mathcal{L}_e + \mathcal{L}_r$  (Table 5). We find that simply sharing the encoders hurts both the entity and relation F1. We think this is because the two tasks have different input formats and require different features for predicting entity types and relations, thus using separate encoders indeed learns better task-specific features. We also explore whether the relation information can improve the entity performance. To do so, we add an auxiliary loss to our entity model, which concatenates the two span representations as well as their element-wise multiplication (see the TEXT variant in Section 5.1) and predicts the relation type between the two spans  $(r \in \mathcal{R}$  or  $\epsilon$ ). Through joint training with this auxiliary relation loss, we observe a negligible improvement  $(< 0.1\%)$  on averaged entity F1 over 5 runs on the ACE05 development set. To summarize, (1) entity information is clearly important in predicting relations (Section 5.1). However, we don't find that relation information to improve our entity model substantially; (2) simply sharing the encoders does not provide benefits to our approach.
5.3 Mitigating Error Propagation
A well-known drawback of pipeline training is the error propagation issue. In our final model, we use gold entities (and their types) to train the relation model and the predicted entities during inference and this may lead to a discrepancy between training and testing. In the following, we describe several attempts we made to address this issue.
We first study whether using predicted entities
<sup>10</sup>Miwa and Bansal (2016) observed a slight improvement on entity F1 by sharing the parameters  $(80.8 \rightarrow 81.8$  F1) on the ACE05 development data. Wadden et al. (2019) observed that their relation propagation layers improved the entity F1 slightly on SciERC but it hurts performance on ACE05.
Figure 2: Effect of different context window sizes, measured on the ACE05 development set with the BERT-base model. We use the same entity model (an entity model with  $W = 300$ ) to report the relation F1 scores (boundaries).
— instead of gold entities — during training can mitigate this issue. We adopt a 10-way jackknifing method, which is a standard technique in many NLP tasks such as dependency parsing (Agić and Schluter, 2017). Specifically, we divide the data into 10 folds and predict the entities in the  $k$ -th fold using an entity model trained on the remainder. As shown in Table 6, we find that jackknifing strategy hurts the final relation performance surprisingly. We hypothesize that it is because it introduced additional noise during training.
Second, we consider using more pairs of spans for the relation model at both training and testing time. The main reason is that in the current pipeline approach, if a gold entity is missed out by the entity model during inference, the relation model will not be able to predict any relations associated with that entity. Following the beam search strategy used in the previous work (Luan et al., 2019; Wadden et al., 2019), we consider using  $\lambda n$  ( $\lambda = 0.4$  and  $n$  is the sentence length) top spans scored by the entity model. We explored several different strategies for encoding the top-scoring spans for the relation model: (1) typed markers: the same as our main model except that we now have markers e.g.,  $\langle S:\epsilon \rangle$ ,  $\langle /S:\epsilon \rangle$  as input tokens; (2) untyped markers: in this case, the relation model is unaware of a span is an entity or not; (3) untyped markers trained with an auxiliary entity loss ( $e \in \mathcal{E}$  or  $\epsilon$ ). As Table 6 shows, none of these changes led to significant improvements and using untyped markers is espe
cially worse because the relation model struggles to identify whether a span is an entity or not.
In sum, we do not find any of these attempts improved performance significantly and our simple pipelined training turns out to be a surprisingly effective strategy. We do not argue that this error propagation issue does not exist or cannot be solved, while we will need to explore better solutions to address this issue.
5.4 Effect of Cross-sentence Context
In Table 1, we demonstrated the improvements from using cross-sentence context on both the entity and relation performance. We explore the effect of different context sizes  $W$  in Figure 2. We find that using cross-sentence context clearly improves both entity and relation F1. However, we find the relation performance doesn't further increase from  $W = 100$  to  $W = 300$ . In our final models, we use  $W = 300$  for the entity model and  $W = 100$  for the relation model.
In this paper, we present a simple and effective approach for end-to-end relation extraction. Our model learns two encoders for entity recognition and relation extraction independently and our experiments show that it outperforms previous state-of-the-art on three standard benchmarks considerably. We conduct extensive analyses to understand the superior performance of our approach and validate the importance of learning distinct contextual representations for entities and relations and using entity information as input features for the relation model. We also propose an efficient approximation, obtaining a large speedup at inference time with a small reduction in accuracy. We hope that this simple model will serve as a very strong baseline and make us rethink the value of joint training in end-to-end relation extraction.
We thank Yi Luan for the help with the datasets and evaluation. We thank Howard Chen, Ameet Deshpande, Dan Friedman, Karthik Narasimhan, and the anonymous reviewers for their helpful comments and feedback. This work is supported in part by a Graduate Fellowship at Princeton University.
This pruning strategy achieves a recall of  $96.7\%$  of gold relations on the development set of ACE05.
Zeljko Agić and Natalie Schluter. 2017. How (not) to train a dependency parser: The curious case of jackknifing part-of-speech taggers. In Association for Computational Linguistics (ACL), pages 679-684.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 2830-2836.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Empirical Methods in Natural Language Processing (EMNLP), pages 3606-3611.
Razvan Bunescu and Raymond Mooney. 2005. A shortest path dependency kernel for relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 724-731.
Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In Association for Computational Linguistics: Human Language Technologies (ACL-HLT), pages 551-560.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171-4186.
Kalpit Dixit and Yaser Al-Onaizan. 2019. Spanlevel model for relation extraction. In Association for Computational Linguistics (ACL), pages 5308-5314.
Radu Florian, Hany Hassan, Abraham Ittycheriah, Hongyan Jing, Nanda Kambhatla, Xiaoqiang Luo, H Nicolov, and Salim Roukos. 2004. A statistical model for multilingual entity detection and tracking. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1-8.
Radu Florian, Hongyan Jing, Nanda Kambhatla, and Imed Zitouni. 2006. Factorizing complex models: A case study in mention detection. In Association for Computational Linguistics (ACL), pages 473-480.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. GraphRel: Modeling text as relational graphs for joint entity and relation extraction. In Association for Computational Linguistics (ACL), pages 1409-1418.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Association for Computational Linguistics (ACL), pages 364-369.
Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for information extraction. In Association for Computational Linguistics (ACL), pages 178-181.
Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Association for Computational Linguistics (ACL), pages 917-928.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR).
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 188-197.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Association for Computational Linguistics (ACL), pages 402-412.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Association for Computational Linguistics (ACL), pages 1340-1350.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Association for Computational Linguistics (ACL).
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Empirical Methods in Natural Language Processing (EMNLP), pages 3219-3232.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 3036-3046.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Association for Computational Linguistics (ACL), pages 1105-1116.
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1858-1869.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In *Empirical Methods in Natural Language Processing (EMNLP)*, pages 43-54.