SlowGuess/NAACL_2021_OCR huggingface dataset:
Yubo Chen†, Yunqi Zhang†, Changran Hu‡, Yongfeng Huang†
$^{\dagger}$ Department of Electronic Engineering & BNRist, Tsinghua University, Beijing, China
$^{\ddagger}$ University of California, Berkeley, USA
ybch14@gmail.com zhang-yq15@outlook.com
changran_hu@berkeley.edu yfhuang@tsinghua.edu.cn
Relational triple extraction is a crucial task for knowledge graph construction. Existing methods mainly focused on explicit relational triples that are directly expressed, but usually suffer from ignoring implicit triples that lack explicit expressions. This will lead to serious incompleteness of the constructed knowledge graphs. Fortunately, other triples in the sentence provide supplementary information for discovering entity pairs that may have implicit relations. Also, the relation types between the implicitly connected entity pairs can be identified with relational reasoning patterns in the real world. In this paper, we propose a unified framework to jointly extract explicit and implicit relational triples. To explore entity pairs that may be implicitly connected by relations, we propose a binary pointer network to extract overlapping relational triples relevant to each word sequentially and retain the information of previously extracted triples in an external memory. To infer the relation types of implicit relational triples, we propose to introduce real-world relational reasoning patterns in our model and capture these patterns with a relation network. We conduct experiments on several benchmark datasets, and the results prove the validity of our method.
Relational triple extraction is defined as automatically recognizing semantic relations with triple structures (subject, relation, object) among multiple entities in a sentence. It is a critical task for constructing Knowledge Graphs (KGs) from unlabeled corpus (Dong et al., 2014).
Early work of relational triple extraction applied pipeline methods (Zelenko et al., 2003; Chan and Roth, 2011), which ran entity recognition and relation classification separately. However, such pipeline approaches suffered from error propagation. To address this issue, recent work proposed to jointly extract entity and relations from the text
Figure 1: An example of explicit and implicit relational triples. Italic phrases are key relational expressions corresponding to the explicit relational triples.
with feature-based methods (Yu and Lam, 2010; Li and Ji, 2014; Ren et al., 2017). Afterward, neural network-based models were proposed to eliminate hand-crafted features (Gupta et al., 2016; Zheng et al., 2017). More recently, several methods were proposed to extract overlapping triples, such as tagging-based (Dai et al., 2019; Wei et al., 2020), graph-based (Wang et al., 2018; Fu et al., 2019), copy-based (Zeng et al., 2018, 2019, 2020) and token pair linking models (Wang et al., 2020).
Existing models achieved considerable success on extracting explicit triples which have direct relational expressions in the sentence. However, there are many implicit relational triples that are not explicitly expressed. For example, in Figure 1, the explicit triples are strongly indicated by the key relational phrases, but the implicit relation "Live in" is not expressed explicitly. Unfortunately, existing methods usually ignored implicit triples (Zhu et al., 2019), which will cause serious incompleteness of the constructed KGs and performance degradation of downstream tasks (Angeli and Manning, 2013; Jia et al., 2020; Jun et al., 2020).
Our work is motivated by several observations. First, other relational triples within a sentence provide supplementary information for discovering entity pairs that may have implicit relational connections. For example, in Figure 1, the explicit triples establish a relational connection between "Mark Spencer" and "Huntsville" through the intermediate entity "Digium". Second, the relation
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5694-5703 June 6-11, 2021. ©2021 Association for Computational Linguistics
types of implicit relation triples can be derived through real-world reasoning patterns. For example, in Figure 1, the reasoning pattern "one lives where the company he works for is located" helps identify the type of the implicit triple as "Live in".
In this paper, we propose a unified framework for the joint extraction of explicit and implicit relational triples. We propose a Binary Pointer Network (BPtrNet), which is based on the pointer network (Vinyals et al., 2015), to extract overlapping relational triples relevant to each word sequentially. To discover implicitly connected entity pairs, we preserve the information of previously extracted triples in an external memory and use it to enhance the extraction of later time steps. To infer the relation types between the implicitly connected entity pairs, we propose to augment our model with real-world relational reasoning patterns and capture the relational inference logic with a Relation Network (RN) (Santoro et al., 2017). The RN obtains a pattern-enhanced representation from the memory for each word pair. Then the Reasoning pattern enhanced BPtrNet (R-BPtrNet) uses the word pair representation to compute a binary score for each candidate triple. Finally, triples with positive scores are output as the extraction result.
- We propose a unified framework to jointly extract explicit and implicit relational triples.
- To discover entity pairs that are implicitly connected by relations, we propose a BPtrNet model to extract overlapping relational triples sequentially and utilize an internal memory to retain the extracted triples.
- To enhance the relation type inference of implicitly connected entity pairs, we propose to introduce relational reasoning patterns, captured with a RN, to augment our model.
- We conduct experiments on several benchmark datasets and the experimental results demonstrate the validity of our method.
Early work of relational triple extraction addressed this task in a pipelined manner (Zelenko et al., 2003; Zhou et al., 2005; Chan and Roth, 2011; Gormley et al., 2015). They first ran named entity recognition to identify all entities and then classified relations between all entity pairs. However, these pipelined methods usually suffered from error propagation problem and failed to capture the
interactions between entities and relations.
To overcome these drawbacks, recent research focused on jointly extracting entities and relations, including feature-based models (Yu and Lam, 2010; Li and Ji, 2014; Ren et al., 2017) and neural network-based models (Gupta et al., 2016; Miwa and Bansal, 2016; Zheng et al., 2017). For example, Ren et al. (2017) proposed to jointly embed entities, relations, text features and type labels into two low-dimensional spaces. Miwa and Bansal (2016) proposed a joint model containing two long-short term memories (LSTMs) (Gers et al., 2000) with shared parameters. Zheng et al. (2017) proposed to extract relational triples directly by transforming this task into a sequence tagging problem, whose tags contain the information of entities and the relations they hold. However, they only assigned one label for each word, which means that this method failed to extract overlapping triples. Subsequent work proposed several mechanisms to solve this problem: (1) labeling tagging sequences for words (Dai et al., 2019) or entities (Yu et al., 2019; Wei et al., 2020); (2) transforming the sentence into a graph structure (Wang et al., 2018; Fu et al., 2019); (3) generating triple element sequences with copy mechanism (Zeng et al., 2018, 2019, 2020; Nayak and Ng, 2020); (4) linking token pairs with a handshake tagging scheme (Wang et al., 2020). However, these methods usually ignored implicit relational triples that are not directly expressed in the sentence (Zhu et al., 2019), thus will lead to the incompleteness of the resulting KGs and negatively affect the performance of downstream tasks (Angeli and Manning, 2013; Jia et al., 2020).
Our work is motivated by two observations. First, other triples in the sentence provide supplementary evidence for discovering entity pairs with implicit relational connections. Second, the relation types of the implicit connections need to be identified through real-world reasoning patterns.
In this paper, we propose a unified framework for the joint extraction of explicit and implicit relational triples. We propose a binary pointer network to sequentially extract overlapping relational triples and externally keep the information of predicted triples for exploring implicitly connected entity pairs. We also propose to introduce real-world reasoning patterns in our model to help derive the relation type of implicit triples with a relation network. Experimental results on several benchmark datasets demonstrate the effectiveness of our method.
Figure 2: The overall framework of our approach.
The overall framework of our approach is shown in Figure 2. We introduce the Binary Pointer Network (BPtrNet) and the Relation Network (RN) in Section 3.1 and 3.2 and the details of training and inference in Section 3.3, respectively.
3.1 Binary Pointer Network
Existing methods usually failed to extract implicit relational triples due to the lack of explicit expressions (Zhu et al., 2019). Fortunately, we observe that other triples in the sentence can help discover entity pairs that may have implicit relational connections. For instance, in the sentence "George is Judy's father and David's grandfather", the relation between Judy and David is not explicitly expressed. In this case, if we first extract the explicit triple (Judy, father, George) and keep its information in our model, we can easily establish an implicit connection between Judy and David through George because George is explicitly connected with David by the relational keyword "grandfather". Inspired by this observation, our model extracts relational triples relevant to each word sequentially and keeps all previous triples of this sentence to enhance the extraction at future time steps. This word-by-word extraction process can be regarded as transforming a text sequence into a sequence of extracting actions, which leads us to a
sequence-to-sequence (seq2seq) model.
Therefore, we propose a Binary Pointer Network (BPtrNet), based on a seq2seq pointer network (Vinyals et al., 2015), to jointly extract explicit and implicit relational triples. Our model first encodes the words of a sentence into vector representations (Section 3.1.1). Then, we use a binary decoder to sequentially transform the vectors into (overlapping) relational triples (Section 3.1.2). We also introduce an external memory to retain previously extracted triples for enhancing future decoding steps (Section 3.1.3).
Given a sentence  $[w_{1},\ldots ,w_{n}]$ , we first capture morphological patterns of entities with a convolutional neural network (CNN) (LeCun et al., 1989) and compute the character representation  $\mathbf{c}_i$  of the word  $w_{i}$ $(i = 1,\dots ,n)$ :  $\mathbf{c}_i = \mathrm{CNN}(w_i;\theta)\in \mathbb{R}^{d_c}$ . Then we introduce the context-sensitive representations  $\mathbf{p}_{1:n}$  captured with a pre-trained Language Model (LM) to bring rich semantics and prior knowledge from the large-scale unlabeled corpus. We feed  $\mathbf{c}_i$ ,  $\mathbf{p}_i$  and the word embedding  $\mathbf{w}_i$  into a bidirectional LSTM (BiLSTM) to compute the contextualized word representations  $\mathbf{x}_{1:n}$  and encode the sentence with another BiLSTM:
$$
\mathbf {x} _ {i} = \operatorname {B i L S T M} ^ {i n} ([ \mathbf {w} _ {i}; \mathbf {p} _ {i}; \mathbf {c} _ {i} ]) \in \mathbb {R} ^ {d _ {i n}}. \tag {1}
$$
$$
\mathbf {h} _ {i} ^ {E} = \operatorname {B i L S T M} ^ {E n c} \left(\mathbf {x} _ {i}\right) \in \mathbb {R} ^ {2 d _ {E}}.
$$
3.1.2 Binary Decoder
First, to capture the interactions between entities and relations, we recognize the entities with a span-based entity tagger (Yu et al., 2019; Wei et al., 2020) and transform the tags into vectors as part of the decoder's input (Figure 2). Specifically, we assign each token a start and end tag to indicate whether the current token corresponds to a start or end position of an entity of a certain type:
$$
\mathbf {h} _ {i} ^ {T} = \operatorname {B i L S T M} ^ {T a g} (\mathbf {x} _ {i}) \in \mathbb {R} ^ {d _ {T}}
$$
$$
p (\mathbf {y} _ {i} ^ {s / e}) = \operatorname {s o f t m a x} \left(\mathbf {W} ^ {s / e} \mathbf {h} _ {i} ^ {T} + \mathbf {b} ^ {s / e}\right) \quad (2)
$$
$$
\operatorname {t a g} _ {i} ^ {s / e} = \arg \max  _ {k} p \left(y _ {i} ^ {s / e} = k\right)
$$
where  $(\mathbf{W}^s,\mathbf{b}^s)$  and  $(\mathbf{W}^{e},\mathbf{b}^{e})$  are parameters of the start and end tag classifiers, respectively. Then we obtain the entity tag embedding  $\mathbf{e}_i\in \mathbb{R}^{d_e}$  by averaging the look-up embeddings of the start and end tags. We also capture a global contextual embedding  $\mathbf{g}$  by max pooling over  $\mathbf{h}_{1:n}^{E}$ . Then we adopt a LSTM as the decoder  $(\mathbf{h}_0^D = \mathbf{h}_n^E)$ :
$$
\mathbf {h} _ {i} ^ {D} = \operatorname {L S T M} ^ {D e c} \left(\left[ \mathbf {x} _ {i}; \mathbf {e} _ {i}; \mathbf {g} \right], \mathbf {h} _ {i - 1} ^ {D}\right) \in \mathbb {R} ^ {2 d _ {E}}. \tag {3}
$$
Next, we introduce how to extract relational triples at the  $i$ -th time step. We consider the current word as the object entity, select words as subjects that form triples with the object from all the words of the sentence, and predict the relation types between the subjects and the object. For example, in Figure 2, when the current object is "Huntsville", the model selects "Digium" as the subject and classifies the relation as "Locate in". Thus ("Digium", "Locate in", "Huntsville") is extracted as a relational triple. Multi-token entities are represented with their last words and recovered by finding the nearest start tags of the same type from their last positions. However, the original softmax pointer in (Vinyals et al., 2015) only allows an object to point to one subject, thus fails to extract multiple triples with overlapping objects. To address this issue, we propose a binary pointer, which independently computes a binary score for each subject to form a relational triple with the current object under each relation type. Our method naturally solves the overlapping triple problem by producing multiple positive scores at one step (Figure 2). We formulate the score of the triple  $(w_{j}, r, w_{i})$  as:
$$
s _ {j i} ^ {(r)} = \sigma \left(\mathbf {r} ^ {\top} \rho \left(\mathbf {W} ^ {p t r} \left[ \mathbf {h} _ {j} ^ {E}; \mathbf {h} _ {i} ^ {D} \right] + \mathbf {b} ^ {p t r}\right)\right), \tag {4}
$$
and extract this candidate triple as a relational triple if  $s_{ji}^{(r)}$  is higher than some threshold, such as 0.5
in our model  $(i,j = 1,\dots ,n)$  .  $\sigma$  and  $\rho$  are the sigmoid and tanh functions, respectively.  $\mathbf{r}\in \mathbb{R}^{d_R}$  is the type embedding of the relation  $r$  .  $\mathbf{W}^{ptr}$  and  $\mathbf{b}^{ptr}$  are parameters of the binary pointer.
3.1.3 External Memory
We introduce an external memory  $\mathbf{M}$  to keep the previously extracted triples of the sentence. We first initialize  $\mathbf{M}$  as an empty set. After the decoder's extraction process at the  $i$ -th time step, we represent the extracted triple  $t = (w_{s_t}, r_t, w_i)$  as:
$$
\mathbf {h} _ {t} ^ {M} = \left[ \mathbf {h} _ {s _ {t}} ^ {E}; \mathbf {r} _ {t}; \mathbf {h} _ {i} ^ {E} \right] \in \mathbb {R} ^ {d _ {M}}. \tag {5}
$$
Then we update the memory with the representations of the output triples  $t_1, \ldots, t_{N_i}$ :
$$
\mathbf {M} \leftarrow [ \mathbf {M}; \mathbf {h} _ {t _ {1}} ^ {M}; \dots ; \mathbf {h} _ {t _ {N _ {i}}} ^ {M} ] \in \mathbb {R} ^ {N \times d _ {M}} \tag {6}
$$
where  $N_{i}$  is the number of the currently extracted triples and  $N = \sum_{k=1}^{i} N_{k}$ . Note that we set and update the external memory for each sentence independently, and the memory stores only the triple representations of one single sentence. Thus triples of other sentences will not be introduced into the sentence currently being extracted. Finally, the triples in the memory are utilized to obtain the reasoning pattern-enhanced representations for future time steps, as described in Section 3.2.
3.2 Relation Network for capturing patterns of relational reasoning
Relation types of implicit relational triples are difficult to infer due to the lack of explicit evidence, thus need to be derived with real-world relational reasoning patterns. For example, in the sentence "George is Judy's father and David's grandfather", the relation type between "Judy" and "David" can be inferred as "father" using the pattern "father's father is called grandfather".
Based on this fact, we propose to enhance our model by introducing real-world relational reasoning patterns. We capture the patterns with a Relation Network (RN) (Santoro et al., 2017), a neural network module specially designed for relational reasoning. A RN is essentially a composite function over a relational triple set  $\mathcal{T}$ :  $\mathrm{RN}(\mathcal{T}) = f_{\phi}(\{\mathbf{g}_{\theta}(t)\}_{t\in \mathcal{T}})$ , where  $f_{\phi}$  is an aggregation function and  $\mathbf{g}_{\theta}$  projects a triple into a fixed-size embedding. We set the memory  $\mathbf{M}$  as the input relational triple set  $\mathcal{T}$  and utilize the RN to learn a pattern-enhanced representation  $\mathbf{h}_{ji}^{P}$  for the word pair  $(w_{j},w_{i})$  at the  $i$ -th time step. First, the  $\mathbf{g}_{\theta}$  reads
the triple representations from  $\mathbf{M}$  and projects them with a fully-connected layer:
$$
\mathbf {g} _ {\theta} (t) = \rho \left(\mathbf {W} ^ {\theta} \mathbf {h} _ {t} ^ {M} + \mathbf {b} ^ {\theta}\right) \in \mathbb {R} ^ {d _ {P}}. \tag {7}
$$
Then  $f_{\phi}$  selects useful triples with a gating network:  $u_t^\phi = \sigma \big(\mathbf{g}_\theta (t)\mathbf{U}^\phi [\mathbf{h}_j^E;\mathbf{h}_i^D ]\big)\in \mathbb{R},$  and aggregates the selected triples with the word pair to compute  $\mathbf{h}_{ji}^{P}$  using another fully-connected layer:
$$
\begin{array}{l} \mathbf {h} _ {j i} ^ {P} = f _ {\phi} (\{\mathbf {g} _ {\theta} (t) \} _ {t \in \mathbf {M}}) \\ = \rho \left(\mathbf {W} ^ {\phi} \left[ \mathbf {h} _ {j} ^ {E}; \mathbf {h} _ {i} ^ {D}; \sum_ {t} u _ {t} ^ {\phi} \mathbf {g} _ {\theta} (t) \right] + \mathbf {b} ^ {\phi}\right). \tag {8} \\ \end{array}
$$
Finally, we modify Equation 4 as  $s_{ji}^{(r)} = \sigma (\mathbf{r}^\top \mathbf{h}_{ji}^P)$  to compute the binary scores of candidate triples. We denote our Reasoning pattern enhanced BPtrNet model as R-BPtrNet. Note that we use quite simple formulas for  $f_{\phi}$  and  $\mathbf{g}_{\theta}$  because our contribution focuses on the effectiveness of introducing relational reasoning patterns for this task rather than the model structure. Exploration for more complex structures will be left for future work.
We calculate the triple loss of a sentence as a binary cross entropy over valid candidate triples  $\mathcal{T}_v$ , whose subject and object are different entities (or the end words of different entities):
$$
\mathcal {L} _ {t} = - \frac {1}{| \mathcal {T} _ {v} |} \left(\sum_ {t \in \mathcal {T} _ {v}} y _ {t} \log s _ {t} + (1 - y _ {t}) \log (1 - s _ {t})\right) \tag {9}
$$
where  $s_t$  is the score of the candidate triple  $t$ ,  $y_t = 1$  for gold triples and 0 for others. We also train the entity tagger with a cross-entropy loss:
$$
\mathcal {L} _ {e} = - \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {* \in \{s, e \}} \log p \left(y _ {i} ^ {*} = \hat {y} _ {i} ^ {*}\right) \tag {10}
$$
where  $\hat{y}_i^{s / e}$  are the gold start and end tags of the  $i$ -th word, respectively. Finally, we train the R-BPtrNet with the joint loss  $\mathcal{L} = \mathcal{L}_t + \mathcal{L}_e$
To prevent error propagation, we use the gold entity tags to filter out valid candidate triples and compute the tag embeddings  $\mathbf{e}_{1:n}$  during training. We also update the memory  $\mathbf{M}$  with the gold relational triples. During inference, we extract triples from scratch and use the predicted entity tags and relational triples instead of the gold ones.
We evaluate our method on two benchmark datasets. NYT (Riedel et al., 2010) consists of sentences from the New York Times corpus and contains 24 relation types. WebNLG (Gardent et al., 2017) was created for natural language generation task. It contains 171 relation types $^2$  and was adopted for relational triple extraction by (Zeng et al., 2018). We split the sentences into three categories: Normal, SingleEntityOverlap (SPO) and EntityPairOverlap (EPO) following Zeng et al. (2018). The statistics of the two datasets are shown in Table 1. Following previous work (Zeng et al., 2018; Wei et al., 2020; Wang et al., 2020), an extracted relational triple is regarded as correct only if the relation and the heads of both subject and object are all correct. We report the standard micro precision, recall, and  $F_{1}$ -scores on both datasets.
Dataset	NYT	WebNLG
Train	Test	Train	Test
Normal	37013	3266	1596	246
SEO	9782	1297	227	457
EPO	14735	978	3406	26
ALL	56195	5000	5019	703
Table 1: Statistics of evaluation datasets.
We determine the hyper-parameters on the validation sets. We use the pre-trained GloVe (Pennington et al., 2014) embeddings as w. We adopt a one-layer CNN with  $d_{c} = 30$  channels to learn c from 30-dimensional randomly-initialized character embeddings. We choose the state-of-the-art RoBERTaLARGE (Liu et al., 2019) model as the pre-tained LM. For a fair comparison with previous methods, we also conduct experiments and report the scores with BERTBASE (Devlin et al., 2019). We set  $d_{in}$  (Equation 1) as 300. The hidden dimensions of the encoder  $d_{E}$  and the entity tagger  $d_{T}$  are both 200. The dimensions of entity tag embeddings  $d_{e}$  and relation type embeddings  $d_{R}$  are set as 50 and 200, respectively. The projection dimension  $d_{P}$  of the RN is set as 500.
2 As mentioned in (Wang et al., 2020), this number is miswritten as 246 in (Wei et al., 2020) and (Yu et al., 2019). Here we quote the correct number from (Wang et al., 2020).
<sup>1</sup>We don't use the more common attention mechanism (Bahdanau et al., 2015) to select triples because the attention weights are restricted to sum to 1. If all triples in the memory are useless, they will still be assigned a large weight due to the restriction, which will confuse the model.
Method	NYT	WebNLG
Prec.	Rec.	F1	Prec.	Rec.	F1
NovelTagging (Zheng et al., 2017)	62.4	31.7	42.0	52.5	19.3	28.3
CopyRE (Zeng et al., 2018)	72.8	69.4	71.1	60.9	61.1	61.0
CopyRERL (Zeng et al., 2019)	77.9	67.2	72.1	63.3	59.9	61.6
GraphRel (Fu et al., 2019)	63.9	60.0	61.9	44.7	41.1	42.9
ETL-Span (Yu et al., 2019)	84.9	72.3	78.1	84.0	91.5	87.6
CopyMTL (Zeng et al., 2020)	75.7	68.7	72.0	58.0	54.9	56.4
WDec (Nayak and Ng, 2020)	94.5	76.2	84.4	-	-	-
CGTUniLM (Ye et al., 2020)	94.7	84.2	89.1	92.9	75.6	83.4
CASREL LSTM (Wei et al., 2020)	84.2	83.0	83.6	86.9	80.6	83.7
CASRELBERT (Wei et al., 2020)	89.7	89.5	89.6	93.4	90.1	91.7
TPLinker LSTM (Wang et al., 2020)	83.8	83.4	83.6	90.8	90.3	90.5
TPLinkerBERT (Wang et al., 2020)	91.3	92.5	91.9	91.8	92.0	91.9
R-BPtrNet	90.9	91.3	91.1	90.7	94.6	92.6
R-BPtrNetBERT	92.7	92.5	92.6	93.7	92.8	93.3
R-BPtrNetRobERTa	94.0	92.9	93.5	94.3	93.3	93.8
Table 2: Performance of our method and previous state-of-the-art models on the NYT and WebNLG test sets. The best scores are in bold and the second-best scores are underlined. R-BPtrNet is our model without pre-trained LMs. R-BPtrNet  $BERT$  and R-BPtrNet  $RoBERTa$  are models using BERT $_{\text{BASE}}$  and RoBERT $_{\text{LARGE}}$ , respectively.