SlowGuess/NAACL_2021_OCR huggingface dataset:
Zexuan Zhong Danqi Chen
{zzhong, danqic}@cs.princeton.edu
End-to-end relation extraction aims to identify named entities and extract relations between them. Most recent work models these two subtasks jointly, either by casting them in one structured prediction framework, or performing multi-task learning through shared representations. In this work, we present a simple pipelined approach for entity and relation extraction, and establish the new state-of-the-art on standard benchmarks (ACE04, ACE05 and SciERC), obtaining a  $1.7\% - 2.8\%$  absolute improvement in relation F1 over previous joint models with the same pre-trained encoders. Our approach essentially builds on two independent encoders and merely uses the entity model to construct the input for the relation model. Through a series of careful examinations, we validate the importance of learning distinct contextual representations for entities and relations, fusing entity information early in the relation model, and incorporating global context. Finally, we also present an efficient approximation to our approach which requires only one pass of both entity and relation encoders at inference time, achieving an  $8 - 16\times$  speedup with a slight reduction in accuracy. $^{1}$
Extracting entities and their relations from unstructured text is a fundamental problem in information extraction. This problem can be decomposed into two subtasks: named entity recognition (Sang and De Meulder, 2003; Ratinov and Roth, 2009) and relation extraction (Zelenko et al., 2002; Bunescu and Mooney, 2005). Early work employed a pipelined approach, training one model to extract entities (Florian et al., 2004, 2006), and another model to classify relations between them (Zhou et al., 2005; Kambhatla, 2004; Chan and Roth, 2011). More recently, however, end-to-end evaluations have been dominated by systems
that model these two tasks jointly (Li and Ji, 2014; Miwa and Bansal, 2016; Katiyar and Cardie, 2017; Zhang et al., 2017a; Li et al., 2019; Luan et al., 2018, 2019; Wadden et al., 2019; Lin et al., 2020; Wang and Lu, 2020). There has been a long held belief that joint models can better capture the interactions between entities and relations and help mitigate error propagation issues.
In this work, we re-examine this problem and present a simple approach which learns two encoders built on top of deep pre-trained language models (Devlin et al., 2019; Beltagy et al., 2019; Lan et al., 2020). The two models — which we refer them as to the entity model and relation model throughout the paper — are trained independently and the relation model only relies on the entity model to provide input features. Our entity model builds on span-level representations and our relation model builds on contextual representations specific to a given pair of spans. Despite its simplicity, we find this pipelined approach to be extremely effective: using the same pre-trained encoders, our model outperforms all previous joint models on three standard benchmarks: ACE04, ACE05 and SciERC, advancing the previous state-of-the-art by  $1.7\% - 2.8\%$  absolute in relation F1.
To better understand the effectiveness of this approach, we carry out a series of careful analyses. We observe that, (1) the contextual representations for the entity and relation models essentially capture distinct information, so sharing their representations hurts performance; (2) it is crucial to fuse the entity information (both boundary and type) at the input layer of the relation model; (3) leveraging cross-sentence information is useful in both tasks. Hence, we expect that this simple model will serve as a very strong baseline in end-to-end relation extraction and make us rethink the value of joint modeling of entities and relations.
Finally, one possible shortcoming of our approach is that we need to run our relation model
1Our code and models are publicly available at https: //github.com/princeton-nlp/PURE.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50-61 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) Entity model
Figure 1: An example from the SciERC dataset (Luan et al., 2018). Given an input sentence  $MORPA$  is a fully implemented parser for a text-to-speech system, an end-to-end relation extraction system is expected to extract that MORPA and  $\mathsf{PARSER}$  are entities of type METHOD, TEXT-TO-SPEECH is a TASK, as well as MORPA is a hyponym of  $\mathsf{PARSER}$  and MORPA is used for TEXT-TO-SPEECH. (a) Our entity model predicts all the entities at once. (b) Our relation model considers every pair of entities independently by inserting typed entity markers (e.g., [S:MD]: the subject is a METHOD, [O:TK]: the object is a TASK). (c) We also proposed an approximation relation model which supports batch computations. The tokens of the same color share the positional embeddings (see Section 4.3 for more details).
once for every pair of entities. To alleviate this issue, we present a novel and efficient alternative by approximating and batching the computations for different groups of entity pairs at inference time. This approximation achieves an  $8 - 16 \times$  speedup with only a slight reduction in accuracy (e.g.,  $1.0\%$  F1 drop on ACE05), which makes our model fast and accurate to use in practice. Our final system is called PURE (the Princeton University Relation Extraction system) and we make our code and models publicly available for the research community.
- We present a simple and effective approach for end-to-end relation extraction, which learns two independent encoders for entity recognition and relation extraction. Our model establishes the new state-of-the-art on three standard benchmarks and surpasses all previous joint models.
- We conduct careful analyses to understand why our approach performs so well and how different factors impact the final performance. We conclude that it is more effective to learn distinct contextual representations for entities and relations than to learn them jointly.
- To speed up the inference time of our model, we also propose a novel efficient approximation, which achieves a large runtime improvement with only a small accuracy drop.
Traditionally, extracting relations between entities in text has been studied as two separate tasks: named entity recognition and relation extraction. In the last several years, there has been a surge of interest in developing models for joint extraction of entities and relations (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016). We group existing joint models into two categories: structured prediction and multi-task learning:
Structured prediction Structured prediction approaches cast the two tasks into one unified framework, although it can be formulated in various ways. Li and Ji (2014) propose an action-based system which identifies new entities as well as links to previous entities, Zhang et al. (2017a); Wang and Lu (2020) adopt a table-filling approach proposed in (Miwa and Sasaki, 2014); Katiyar and Cardie (2017) and Zheng et al. (2017) employ sequence tagging-based approaches; Sun et al. (2019) and Fu et al. (2019) propose graph-based approaches to jointly predict entity and relation types; and, Li et al. (2019) convert the task into a multi-turn question answering problem. All of these approaches need to tackle a global optimization problem and perform joint decoding at inference time, using beam search or reinforcement learning.
Multi-task learning This family of models essentially builds two separate models for entity
recognition and relation extraction and optimizes them together through parameter sharing. Miwa and Bansal (2016) propose to use a sequence tagging model for entity prediction and a tree-based LSTM model for relation extraction. The two models share one LSTM layer for contextualized word representations and they find sharing parameters improves performance (slightly) for both models. The approach of Bekoulis et al. (2018) is similar except that they model relation classification as a multi-label head selection problem. Note that these approaches still perform pipelined decoding: entities are first extracted and the relation model is applied on the predicted entities.
The closest work to ours is DYGIE and DYGIE++ (Luan et al., 2019; Wadden et al., 2019), which builds on recent span-based models for coreference resolution (Lee et al., 2017) and semantic role labeling (He et al., 2018). The key idea of their approaches is to learn shared span representations between the two tasks and update span representations through dynamic graph propagation layers. A more recent work Lin et al. (2020) further extends DYGIE++ by incorporating global features based on cross-subtask and cross-instance constraints. Our approach is much simpler and we will detail the differences in Section 3.2 and explain why our model performs better.
In this section, we first formally define the problem of end-to-end relation extraction in Section 3.1 and then detail our approach in Section 3.2. Finally, we present our approximation solution in Section 3.3, which considerably improves the efficiency of our approach during inference.
The input of the problem is a sentence  $X$  consisting of  $n$  tokens  $x_{1}, x_{2}, \ldots, x_{n}$ . Let  $S = \{s_{1}, s_{2}, \ldots, s_{m}\}$  be all the possible spans in  $X$  of up to length  $L$  and  $\mathrm{START}(i)$  and  $\mathrm{END}(i)$  denote start and end indices of  $s_{i}$ . Optionally, we can incorporate cross-sentence context to build better contextual representations (Section 3.2). The problem can be decomposed into two sub-tasks:
Named entity recognition Let  $\mathcal{E}$  denote a set of pre-defined entity types. The named entity recognition task is, for each span  $s_i\in S$  , to predict an
entity type  $y_{e}(s_{i}) \in \mathcal{E}$  or  $y_{e}(s_{i}) = \epsilon$  representing span  $s_i$  is not an entity. The output of the task is  $Y_{e} = \{(s_{i}, e) : s_{i} \in S, e \in \mathcal{E}\}$ .
Relation extraction Let  $\mathcal{R}$  denote a set of predefined relation types. The task is, for every pair of spans  $s_i\in S,s_j\in S$  , to predict a relation type  $y_{r}(s_{i},s_{j})\in \mathcal{R}$  , or there is no relation between them:  $y_{r}(s_{i},s_{j}) = \epsilon$  . The output of the task is  $Y_{r} = \{(s_{i},s_{j},r):s_{i},s_{j}\in S,r\in \mathcal{R}\}$
As shown in Figure 1, our approach consists of an entity model and a relation model. The entity model first takes the input sentence and predicts an entity type (or  $\epsilon$ ) for each single span. We then process every pair of candidate entities independently in the relation model by inserting extra marker tokens to highlight the subject and object and their types. We will detail each component below, and finally summarize the differences between our approach and DYGIE++ (Wadden et al., 2019).
Entity model Our entity model is a standard span-based model following prior work (Lee et al., 2017; Luan et al., 2018, 2019; Wadden et al., 2019). We first use a pre-trained language model (e.g., BERT) to obtain contextualized representations  $\mathbf{x}_t$  for each input token  $x_t$ . Given a span  $s_i \in S$ , the span representation  $\mathbf{h}_e(s_i)$  is defined as:
$$
\mathbf {h} _ {e} (s _ {i}) = [ \mathbf {x} _ {\mathrm {S T A R T} (i)}; \mathbf {x} _ {\mathrm {E N D} (i)}; \boldsymbol {\phi} (s _ {i}) ],
$$
where  $\phi (s_i)\in \mathbb{R}^{d_F}$  represents the learned embeddings of span width features. The span representation  $\mathbf{h}_e(s_i)$  is then fed into a feedforward network to predict the probability distribution of the entity type  $e\in \mathcal{E}\cup \{\epsilon \} :P_e(e\mid s_i)$
Relation model The relation model aims to take a pair of spans  $s_i, s_j$  (a subject and an object) as input and predicts a relation type or  $\epsilon$ . Previous approaches (Luan et al., 2018, 2019; Wadden et al., 2019) re-use the span representations  $\mathbf{h}_e(s_i), \mathbf{h}_e(s_j)$  to predict the relationship between  $s_i$  and  $s_j$ . We hypothesize that these representations only capture contextual information around each individual entity and might fail to capture the dependencies between the pair of spans. We also argue that sharing the contextual representations between different pairs of spans may be suboptimal. For instance, the words is  $a$  in Figure 1 are crucial in understanding the relationship between MORPA and PArSER but not for MORPA and TEXT-TO-SPEECH.
2This is an orthogonal contribution to ours and we will explore it for future work.
Our relation model instead processes each pair of spans independently and inserts typed markers at the input layer to highlight the subject and object and their types. Specifically, given an input sentence  $X$  and a pair of subject-object spans  $s_i, s_j$ , where  $s_i, s_j$  have a type of  $e_i, e_j \in \mathcal{E} \cup \{\epsilon\}$  respectively. We define text markers as  $\langle \mathrm{S}: e_i \rangle$ ,  $\langle \mathrm{S}: e_j \rangle$ ,  $\langle \mathrm{O}: e_j \rangle$ , and  $\langle \mathrm{O}: e_j \rangle$ , and insert them into the input sentence before and after the subject and object spans (Figure 1 (b)).<sup>3</sup> Let  $\widehat{X}$  denote this modified sequence with text markers inserted:
$$
\begin{array}{l} \widehat {X} = \ldots \langle \mathrm {S}: e _ {i} \rangle , x _ {\mathrm {S T A R T} (i)}, \ldots , x _ {\mathrm {E N D} (i)}, \langle / \mathrm {S}: e _ {i} \rangle , \\ \dots \langle \mathrm {O}: e _ {j} \rangle , x _ {\mathrm {S T A R T} (j)}, \dots , x _ {\mathrm {E N D} (j)}, \langle / \mathrm {O}: e _ {j} \rangle , \dots . \\ \end{array}
$$
We apply a second pre-trained encoder on  $\widehat{X}$  and denote the output representations by  $\widehat{\mathbf{x}}_t$ . We concatenate the output representations of two start positions and obtain the span-pair representation:
$$
\mathbf {h} _ {r} (s _ {i}, s _ {j}) = [ \widehat {\mathbf {x}} _ {\text {S T A R T} (i)}; \widehat {\mathbf {x}} _ {\text {S T A R T} (j)} ],
$$
where  $\widehat{\mathrm{START}}(i)$  and  $\widehat{\mathrm{START}}(j)$  are the indices of  $\langle \mathbf{S}:e_i\rangle$  and  $\langle \mathbf{O}:e_j\rangle$  in  $\widehat{X}$ . Finally, the representation  $\mathbf{h}_r(s_i,s_j)$  will be fed into a feedforward network to predict the probability distribution of the relation type  $r\in \mathcal{R}\cup \{\epsilon\} :P_r(r|s_i,s_j)$ .
This idea of using additional markers to highlight the subject and object is not entirely new as it has been studied recently in relation classification (Zhang et al., 2019; Soares et al., 2019; Peters et al., 2019). However, most relation classification tasks (e.g., TACRED (Zhang et al., 2017b)) only focus on a given pair of subject and object in an input sentence and its effectiveness has not been evaluated in the end-to-end setting in which we need to classify the relationships between multiple entity mentions. We observed a large improvement in our experiments (Section 5.1) and this strengthens our hypothesis that modeling the relationship between different entity pairs in one sentence require different contextual representations. Furthermore, Zhang et al. (2019); Soares et al. (2019) only consider untyped markers (e.g.,  $\langle S\rangle$ ,  $\langle /S\rangle$ ) and previous end-to-end models (e.g., (Wadden et al., 2019)) only inject the entity type information into the relation model through auxiliary losses. We find that injecting type information at the input layer is very helpful in distinguishing entity types — for example, whether
"Disney" refers to a person or an organization—before trying to understand the relations.
Cross-sentence context Cross-sentence information can be used to help predict entity types and relations, especially for pronominal mentions. Luan et al. (2019); Wadden et al. (2019) employ a propagation mechanism to incorporate cross-sentence context. Wadden et al. (2019) also add a 3-sentence context window which is shown to improve performance. We also evaluate the importance of leveraging cross-sentence context in our approach. As we expect that pre-trained language models to be able to capture long-range dependencies, we simply incorporate cross-sentence context by extending the sentence to a fixed window size  $W$  for both the entity and relation model. Specifically, given an input sentence with  $n$  words, we augment the input with  $(W - n) / 2$  words from the left context and right context respectively.
Training & inference For both entity model and relation model, we fine-tune the two pre-trained language models using task-specific losses. We use cross-entropy loss for both models:
$$
\begin{array}{l} \mathcal {L} _ {e} = - \sum_ {s _ {i} \in S} \log P _ {e} \left(e _ {i} ^ {*} \mid s _ {i}\right) \\ \mathcal{L}_{r} = -\sum_{s_{i},s_{j}\in S_{G},s_{i}\neq s_{j}}\log P_{r}(r^{*}_{i,j}\mid s_{i},s_{j}), \\ \end{array}
$$
where  $e_i^*$  represents the gold entity type of  $s_i$  and  $r_{i,j}^{*}$  represents the gold relation type of span pair  $s_i, s_j$  in the training data. For training the relation model, we only consider the gold entities  $S_G \subset S$  in the training set and use the gold entity labels as the input of the relation model. We considered training on predicted entities as well as all spans  $S$  (with pruning), but none of them led to meaningful improvements compared to this simple pipelined training (see more discussion in Section 5.3). During inference, we first predict the entities by taking  $y_e(s_i) = \arg \max_{e \in \mathcal{E} \cup \{\epsilon\}} P_e(e|s_i)$ . Denote  $S_{\mathrm{pred}} = \{s_i : y_e(s_i) \neq \epsilon\}$ , we enumerate all the spans  $s_i, s_j \in S_{\mathrm{pred}}$  and use  $y_e(s_i), y_e(s_j)$  to construct the input for the relation model  $P_r(r \mid s_i, s_j)$ .
Differences from DYGIE++ Our approach differs from DYGIE++ (Luan et al., 2019; Wadden et al., 2019) in the following ways: (1) We use separate encoders for the entity and relation models, without any multi-task learning. The predicted entity types are used directly to construct the input for the relation model. (2) The contextual repre
<sup>3</sup>Our final model indeed only considers  $e_i, e_j \neq \epsilon$ . We have explored strategies using spans which are predicted as  $\epsilon$  for the relation model but didn't find improvement. See Section 5.3 for more discussion.
sentations in the relation model are specific to each pair of spans by using the text markers. (3) We only incorporate cross-sentence information by extending the input with additional context (as they did) and we do not employ any graph propagation layers and beam search. As a result, our model is much simpler. As we will show in the experiments (Section 4), it also achieves large gains in all the benchmarks, using the same pre-trained encoders.
3.3 Efficient Batch Computations
One possible shortcoming of our approach is that we need to run our relation model once for every pair of entities. To alleviate this issue, we propose a novel and efficient alternative to our relation model. The key problem is that we would like to re-use computations for different pairs of spans in the same sentence. This is impossible in our original model because we must insert the entity markers for each pair of spans independently. To this end, we propose an approximation model by making two major changes to the original relation model. First, instead of directly inserting entity markers into the original sentence, we tie the position embeddings of the markers with the start and end tokens of the corresponding span:
$$
\mathrm {P} \left(\langle \mathrm {S}: e _ {i} \rangle , \mathrm {P} \left(\langle / \mathrm {S}: e _ {i} \rangle\right) := \mathrm {P} \left(x _ {\text {S T A R T} (i)}\right), \mathrm {P} \left(x _ {\text {E N D} (i)}\right) \right.
$$
$$
\mathrm {P} (\langle \mathrm {O}: e _ {j} \rangle), \mathrm {P} (\langle / \mathrm {O}: e _ {j} \rangle) := \mathrm {P} (x _ {\mathrm {S T A R T} (j)}), \mathrm {P} (x _ {\mathrm {E N D} (j)}),
$$
where  $\mathrm{P}(\cdot)$  denotes the position id of a token. As the example shown in Figure 1, if we want to classify the relationship between MORPA and PARSER, the first entity marker  $\langle S: \text{METHOD} \rangle$  will share the position embedding with the token MOR. By doing this, the position embeddings of the original tokens will not be changed.
Second, we add a constraint to the attention layers. We enforce the text tokens to only attend to text tokens and not attend to the marker tokens while an entity marker token can attend to all the text tokens and all the 4 marker tokens associated with the same span pair. These two modifications allow us to re-use the computations of all text tokens, because the representations of text tokens are independent of the entity marker tokens. Thus, we can batch multiple pairs of spans from the same sentence in one run of the relation model. In practice, we add all marker tokens to the end of the sentence
to form an input that batches a set of span pairs (Figure 1(c)). This leads to a large speedup at inference time and only a small drop in performance (Section 4.3).
Datasets We evaluate our approach on three popular end-to-end relation extraction datasets: ACE05 $^{5}$ , ACE04 $^{6}$ , and SciERC (Luan et al., 2018). Table 2 shows the data statistics of each dataset. The ACE05 and ACE04 datasets are collected from a variety of domains, such as newswire and online forums. The SciERC dataset is collected from 500 AI paper abstracts and defines scientific terms and relations specially for scientific knowledge graph construction. We follow previous work and use the same preprocessing procedure and splits for all datasets. See Appendix A for more details.
Evaluation metrics We follow the standard evaluation protocol and use micro F1 measure as the evaluation metric. For named entity recognition, a predicted entity is considered as a correct prediction if its span boundaries and the predicted entity type are both correct. For relation extraction, we adopt two evaluation metrics: (1) boundaries evaluation (Rel): a predicted relation is considered as a correct prediction if the boundaries of two spans are correct and the predicted relation type is correct; (2) strict evaluation  $(\mathrm{Rel} + )$  : in addition to what is required in the boundaries evaluation, predicted entity types also must be correct. More discussion of the evaluation settings can be found in Bekoulis et al. (2018); Taille et al. (2020).
Implementation details We use bert-base-uncased (Devlin et al., 2019) and albert-xxlargev1 (Lan et al., 2020) as the base encoders for ACE04 and ACE05, for a fair comparison with previous work and an investigation of small vs large pre-trained models. We also use scibert-scivocab-uncased (Beltagy et al., 2019) as the base encoder for SciERC, as this in-domain pre-trained model is shown to be more effective than BERT (Wadden et al., 2019). We use a context window size of  $W = 300$  for the entity model and  $W = 100$  for
5 catalog.ldc.upenn.edu/LDC2006T06
$^{6}$ catalog.ldc.upenn.edu/LDC2005T09
As detailed in Table 1, some previous work used BERT-large models. We are not able to do a comprehensive study of all the pre-trained models and our BERT-base results are generally higher than most published results using larger models.
<sup>4</sup>They also incorporated coreferences and event prediction in their framework. We focus on entity and relation extraction in this paper and we leave these extensions to future work.