Venue: NAACL 2021

Jointly Extracting Explicit and Implicit Relational Triples with Reasoning Pattern Enhanced Binary Pointer Network

Yubo Chen\({}^{\dagger}\), Yunqi Zhang\({}^{\ddagger}\), Changran Hu\({}^{\ddagger}\), Yongfeng Huang\({}^{\dagger}\)

\({}^{\dagger}\)Department of Electronic Engineering & BNRist, Tsinghua University, Beijing, China

\({}^{\ddagger}\)University of California, Berkeley, USA

ybch14@gmail.com zhang-yd15@outlook.com changran_hu@berkeley.edu yfhuang@tsinghua.edu.cn

###### Abstract

Relational triple extraction is a crucial task for knowledge graph construction. Existing methods mainly focused on explicit relational triples that are directly expressed, but usually suffer from ignoring implicit triples that lack explicit expressions. This will lead to serious incompleteness of the constructed knowledge graphs. Fortunately, other triples in the sentence provide supplementary information for discovering entity pairs that may have implicit relations. Also, the relation types between the implicitly connected entity pairs can be identified with relational reasoning patterns in the real world. In this paper, we propose a unified framework to jointly extract explicit and implicit relational triples. To explore entity pairs that may be implicitly connected by relations, we propose a binary pointer network to extract overlapping relational triples relevant to each word sequentially and retain the information of previously extracted triples in an external memory. To infer the relation types of implicit relational triples, we propose to introduce real-world relational reasoning patterns in our model and capture these patterns with a relation network. We conduct experiments on several benchmark datasets, and the results prove the validity of our method.

+
Footnote †: _Correspondings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5694–5703 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Relational triple extraction is defined as automatically recognizing semantic relations with triple structures (_subject_, _relation_, _object_) among multiple entities in a sentence. It is a critical task for constructing Knowledge Graphs (KGs) from unlabeled corpus Dong et al. (2014).

Early work of relational triple extraction applied pipeline methods Zelenko et al. (2003); Chan and Roth (2011), which ran entity recognition and relation classification separately. However, such pipeline approaches suffered from error propagation. To address this issue, recent work proposed to jointly extract entity and relations from the text with feature-based methods Yu and Lam (2010); Li and Ji (2014); Ren et al. (2017). Afterward, neural network-based models were proposed to eliminate hand-crafted features Gupta et al. (2016); Zheng et al. (2017). More recently, several methods were proposed to extract _overlapping_ triples, such as tagging-based Dai et al. (2019); Wei et al. (2020), graph-based Wang et al. (2018); Fu et al. (2019), copy-based Zeng et al. (2018, 2019, 2020) and token pair linking models Wang et al. (2020).

Existing models achieved considerable success on extracting _explicit_ triples which have direct relational expressions in the sentence. However, there are many _implicit_ relational triples that are not explicitly expressed. For example, in Figure 1, the explicit triples are strongly indicated by the key relational phrases, but the implicit relation "_Live in_" is not expressed explicitly. Unfortunately, existing methods usually ignored implicit triples Zhu et al. (2019), which will cause serious incompleteness of the constructed KGs and performance degradation of downstream tasks Angeli and Manning (2013); Jia et al. (2020); Jun et al. (2020).

Our work is motivated by several observations. First, other relational triples within a sentence provide supplementary information for discovering entity pairs that may have implicit relational connections. For example, in Figure 1, the explicit triples establish a relational connection between "_Mark Spencer_" and "_Huntsville_" through the intermediate entity "_Digium_". Second, the relation

Figure 1: An example of explicit and implicit relational triples. Italic phrases are key relational expressions corresponding to the explicit relational triples.

 

types of implicit relation triples can be derived through real-world reasoning patterns. For example, in Figure 1, the reasoning pattern "one _lives_ where the company he _works_ for is _located_" helps identify the type of the implicit triple as "_Live in_".

In this paper, we propose a unified framework for the joint extraction of explicit and implicit relational triples. We propose a Binary Pointer Network (BPtrNet), which is based on the pointer network Vinyals et al. (2015), to extract overlapping relational triples relevant to each word sequentially. To discover implicitly connected entity pairs, we preserve the information of previously extracted triples in an external memory and use it to enhance the extraction of later time steps. To infer the relation types between the implicitly connected entity pairs, we propose to augment our model with real-world relational reasoning patterns and capture the relational inference logic with a Relation Network (RN) Santoro et al. (2017). The RN obtains a pattern-enhanced representation from the memory for each word pair. Then the Reasoning pattern enhanced BPtrNet (R-BPtrNet) uses the word pair representation to compute a binary score for each candidate triple. Finally, triples with positive scores are output as the extraction result.

The main contributions of this paper are:

* We propose a unified framework to jointly extract explicit and implicit relational triples.
* To discover entity pairs that are implicitly connected by relations, we propose a BPtrNet model to extract overlapping relational triples sequentially and utilize an internal memory to retain the extracted triples.
* To enhance the relation type inference of implicitly connected entity pairs, we propose to introduce relational reasoning patterns, captured with a RN, to augment our model.
* We conduct experiments on several benchmark datasets and the experimental results demonstrate the validity of our method.

## 2 Related Work

Early work of relational triple extraction addressed this task in a pipelined manner Zelenko et al. (2003); Zhou et al. (2005); Chan and Roth (2011); Gormley et al. (2015). They first ran named entity recognition to identify all entities and then classified relations between all entity pairs. However, these pipelined methods usually suffered from error propagation problem and failed to capture the interactions between entities and relations.

To overcome these drawbacks, recent research focused on jointly extracting entities and relations, including feature-based models Yu and Lam (2010); Li and Ji (2014); Ren et al. (2017) and neural network-based models Gupta et al. (2016); Miwa and Bansal (2016); Zheng et al. (2017). For example, Ren et al. (2017) proposed to jointly embed entities, relations, text features and type labels into two low-dimensional spaces. Miwa and Bansal (2016) proposed a joint model containing two long-short term memories (LSTMs) Gers et al. (2000) with shared parameters. Zheng et al. (2017) proposed to extract relational triples directly by transforming this task into a sequence tagging problem, whose tags contain the information of entities and the relations they hold. However, they only assigned one label for each word, which means that this method failed to extract overlapping triples. Subsequent work proposed several mechanisms to solve this problem: (1) labeling tagging sequences for words Dai et al. (2019) or entities Yu et al. (2019); Wei et al. (2020); (2) transforming the sentence into a graph structure Wang et al. (2018); Fu et al. (2019); (3) generating triple element sequences with copy mechanism Zeng et al. (2018, 2019, 2020); Nayak and Ng (2020); (4) linking token pairs with a handshake tagging scheme Wang et al. (2020). However, these methods usually ignored implicit relational triples that are not directly expressed in the sentence Zhu et al. (2019), thus will lead to the incompleteness of the resulting KGs and negatively affect the performance of downstream tasks Angeli and Manning (2013); Jia et al. (2020).

Our work is motivated by two observations. First, other triples in the sentence provide supplementary evidence for discovering entity pairs with implicit relational connections. Second, the relation types of the implicit connections need to be identified through real-world reasoning patterns.

In this paper, we propose a unified framework for the joint extraction of explicit and implicit relational triples. We propose a binary pointer network to sequentially extract overlapping relational triples and externally keep the information of predicted triples for exploring implicitly connected entity pairs. We also propose to introduce real-world reasoning patterns in our model to help derive the relation type of implicit triples with a relation network. Experimental results on several benchmark datasets demonstrate the effectiveness of our method 



## 3 Our Approach

The overall framework of our approach is shown in Figure 2. We introduce the Binary Pointer Network (BPtrNet) and the Relation Network (RN) in Section 3.1 and 3.2 and the details of training and inference in Section 3.3, respectively.

### Binary Pointer Network

Existing methods usually failed to extract implicit relational triples due to the lack of explicit expressions Zhu et al. (2019). Fortunately, we observe that other triples in the sentence can help discover entity pairs that may have implicit relational connections. For instance, in the sentence "_George is Judy's father and David's grandfather_", the relation between Judy and David is not explicitly expressed. In this case, if we first extract the explicit triple (Judy, father, George) and keep its information in our model, we can easily establish an implicit connection between Judy and David through George because George is explicitly connected with David by the relational keyword "_grandfather_". Inspired by this observation, our model extracts relational triples relevant to each word sequentially and keeps all previous triples of this sentence to enhance the extraction at future time steps. This word-by-word extraction process can be regarded as transforming a text sequence into a sequence of extracting actions, which leads us to a sequence-to-sequence (seq2seq) model.

Therefore, we propose a Binary Pointer Network (BPtrNet), based on a seq2seq pointer network Vinyals et al. (2015), to jointly extract explicit and implicit relational triples. Our model first encodes the words of a sentence into vector representations (Section 3.1.1). Then, we use a binary decoder to sequentially transform the vectors into (overlapping) relational triples (Section 3.1.2). We also introduce an external memory to retain previously extracted triples for enhancing future decoding steps (Section 3.1.3).

#### 3.1.1 Encoder

Given a sentence \([w_{1},\dots,w_{n}]\), we first capture morphological patterns of entities with a convolutional neural network (CNN) LeCun et al. (1989) and compute the character representation \(\mathbf{c}_{i}\) of the word \(w_{i}\) (\(i=1,\dots,n\)): \(\mathbf{c}_{i}=\text{CNN}(w_{i};\theta)\in\mathbb{R}^{d_{e}}\). Then we introduce the context-sensitive representations \(\mathbf{p}_{1:n}\) captured with a pre-trained Language Model (LM) to bring rich semantics and prior knowledge from the large-scale unlabeled corpus. We feed \(\mathbf{c}_{i}\), \(\mathbf{p}_{i}\) and the word embedding \(\mathbf{w}_{i}\) into a bidirectional LSTM (BiLSTM) to compute the contextualized word representations \(\mathbf{x}_{1:n}\) and encode the sentence with another BiLSTM:

\[\begin{split}&\mathbf{x}_{i}=\text{BiLSTM}^{in}([\mathbf{ w}_{i};\mathbf{p}_{i};\mathbf{c}_{i}])\in\mathbb{R}^{d_{in}}\\ &\mathbf{h}_{i}^{E}=\text{BiLSTM}^{Enc}(\mathbf{x}_{i})\in \mathbb{R}^{2d_{E}}.\end{split} \tag{1}\]

Figure 2: The overall framework of our approach.





#### 3.1.2 Binary Decoder

First, to capture the interactions between entities and relations, we recognize the entities with a span-based entity tagger Yu et al. (2019); Wei et al. (2020) and transform the tags into vectors as part of the decoder's input (Figure 2). Specifically, we assign each token a start and end tag to indicate whether the current token corresponds to a start or end position of an entity of a certain type:

\[{\bf h}_{i}^{T} =\text{BiLSTM}^{Tag}({\bf x}_{i})\in\mathbb{R}^{d_{T}} \tag{2}\] \[p({\bf y}_{i}^{s/e}) =\text{softmax}({\bf W}^{s/e}{\bf h}_{i}^{T}+{\bf b}^{s/e})\] \[\text{tag}_{i}^{s/e} =\arg\max_{k}p(y_{i}^{s/e}=k)\]

where \(({\bf W}^{s},{\bf b}^{s})\) and \(({\bf W}^{e},{\bf b}^{e})\) are parameters of the start and end tag classifiers, respectively. Then we obtain the entity tag embedding \({\bf e}_{i}\in\mathbb{R}^{d_{e}}\) by averaging the look-up embeddings of the start and end tags. We also capture a global contextual embedding \({\bf g}\) by max pooling over \({\bf h}_{1:n}^{E}\). Then we adopt a LSTM as the decoder (\({\bf h}_{0}^{D}={\bf h}_{n}^{E}\)):

\[{\bf h}_{i}^{D}=\text{LSTM}^{Dec}([{\bf x}_{i};{\bf e}_{i};{\bf g}],{\bf h}_ {i-1}^{D})\in\mathbb{R}^{2d_{E}}. \tag{3}\]

Next, we introduce how to extract relational triples at the \(i\)-th time step. We consider the current word as the object entity, select words as subjects that form triples with the object from all the words of the sentence, and predict the relation types between the subjects and the object. For example, in Figure 2, when the current object is "_Huntsville_", the model selects "_Digium_" as the subject and classifies the relation as "_Locate in_". Thus ("_Digium_", "_Locate in_", "_Huntsville_") is extracted as a relational triple. Multi-token entities are represented with their last words and recovered by finding the nearest start tags of the same type from their last positions. However, the original softmax pointer in Vinyals et al. (2015) only allows an object to point to **one** subject, thus fails to extract multiple triples with overlapping objects. To address this issue, we propose a binary pointer, which independently computes a binary score for each subject to form a relational triple with the current object under each relation type. Our method naturally solves the overlapping triple problem by producing multiple positive scores at one step (Figure 2). We formulate the score of the triple (\(w_{j}\), \(r\), \(w_{i}\)) as:

\[s_{ji}^{(r)}=\sigma({\bf r}^{\top}\rho({\bf W}^{ptr}[{\bf h}_{j}^{E};{\bf h}_ {i}^{D}]+{\bf b}^{ptr})), \tag{4}\]

and extract this candidate triple as a relational triple if \(s_{ji}^{(r)}\) is higher than some threshold, such as \(0.5\) in our model \((i,j=1,\dots,n)\). \(\sigma\) and \(\rho\) are the sigmoid and tanh functions, respectively. \({\bf r}\in\mathbb{R}^{d_{R}}\) is the type embedding of the relation \(r\). \({\bf W}^{ptr}\) and \({\bf b}^{ptr}\) are parameters of the binary pointer.

#### 3.1.3 External Memory

We introduce an external memory \({\bf M}\) to keep the previously extracted triples of the sentence. We first initialize \({\bf M}\) as an empty set. After the decoder's extraction process at the \(i\)-th time step, we represent the extracted triple \(t=(w_{s_{t}},r_{t},w_{i})\) as:

\[{\bf h}_{t}^{M}=[{\bf h}_{s_{t}}^{E};{\bf r}_{t};{\bf h}_{i}^{E}]\in\mathbb{ R}^{d_{M}}. \tag{5}\]

Then we update the memory with the representations of the output triples \(t_{1},\ldots,t_{N_{i}}\):

\[{\bf M}\leftarrow[{\bf M};{\bf h}_{t_{1}}^{M};\dots;{\bf h}_{t_{N_{i}}}^{M}] \in\mathbb{R}^{N\times d_{M}} \tag{6}\]

where \(N_{i}\) is the number of the currently extracted triples and \(N=\sum_{k=1}^{i}N_{k}\). Note that we set and update the external memory for each sentence independently, and the memory stores only the triple representations of one single sentence. Thus triples of other sentences will not be introduced into the sentence currently being extracted. Finally, the triples in the memory are utilized to obtain the reasoning pattern-enhanced representations for future time steps, as described in Section 3.2.

### Relation Network for capturing patterns of relational reasoning

Relation types of implicit relational triples are difficult to infer due to the lack of explicit evidence, thus need to be derived with real-world relational reasoning patterns. For example, in the sentence "_George is Judy's father and David's grandfather_", the relation type between "_Judy_" and "_David_" can be inferred as "_father_" using the pattern "_father_" is called _grandfather_".

Based on this fact, we propose to enhance our model by introducing real-world relational reasoning patterns. We capture the patterns with a Relation Network (RN) Santoro et al. (2017), a neural network module specially designed for relational reasoning. A RN is essentially a composite function over a relational triple set \(\mathcal{T}\): \(\text{RN}(\mathcal{T})=f_{\phi}\big{(}\{{\bf g}_{\theta}(t)\}_{t\in\mathcal{ T}}\big{)},\) where \(f_{\phi}\) is an aggregation function and \({\bf g}_{\theta}\) projects a triple into a fixed-size embedding. We set the memory \({\bf M}\) as the input relational triple set \(\mathcal{T}\) and utilize the RN to learn a pattern-enhanced representation \({\bf h}_{ji}^{P}\) for the word pair (\(w_{j},w_{i}\)) at the \(i\)-th time step. First, the \({\bf g}_{\theta}\) reads

the triple representations from \({\bf M}\) and projects them with a fully-connected layer:

\[\mathbf{g}_{\theta}(t)=\rho(\mathbf{W^{\phi}h}_{t}^{M}+\mathbf{b}^{\theta})\in \mathbb{R}^{d_{P}}. \tag{7}\]

Then \(f_{\phi}\) selects useful triples with a gating network1: \(u_{t}^{\phi}=\sigma\big{(}\mathbf{g}_{\theta}(t)\mathbf{U^{\phi}}[\mathbf{h}_{j }^{E};\mathbf{h}_{i}^{D}]\big{)}\in\mathbb{R}\), and aggregates the selected triples with the word pair to compute \(\mathbf{h}_{ji}^{P}\) using another fully-connected layer:

Footnote 1: We don’t use the more common attention mechanism Bahdanau et al. (2015) to select triples because the attention weights are restricted to sum to 1. If all triples in the memory are useless, they will still be assigned a large weight due to the restriction, which will confuse the model.

\[\begin{split}&\mathbf{h}_{ji}^{P}=f_{\phi}(\{\mathbf{g}_{\theta}( t)\}_{t\in{\bf M}})\\ &=\rho\bigg{(}\mathbf{W^{\phi}}\bigg{[}\mathbf{h}_{j}^{E}; \mathbf{h}_{i}^{D};\sum_{t}u_{t}^{\phi}\mathbf{g}_{\theta}(t)\bigg{]}+\mathbf{ b^{\phi}}\bigg{)}.\end{split} \tag{8}\]

Finally, we modify Equation 4 as \(s_{ji}^{(r)}=\sigma(\mathbf{r}^{\top}\mathbf{h}_{ji}^{P})\) to compute the binary scores of candidate triples. We denote our Reasoning pattern enhanced BPtrNet model as R-BPtrNet. Note that we use quite simple formulas for \(f_{\phi}\) and \(\mathbf{g}_{\theta}\) because our contribution focuses on the effectiveness of introducing relational reasoning patterns for this task rather than the model structure. Exploration for more complex structures will be left for future work.

### Training and Inference

We calculate the triple loss of a sentence as a binary cross entropy over valid candidate triples \(\mathcal{T}_{v}\), whose subject and object are different entities (or the end words of different entities):

\[\mathcal{L}_{t}=-\frac{1}{|\mathcal{T}_{v}|}\bigg{(}\sum_{t\in\mathcal{T}_{v} }y_{t}\log s_{t}+(1-y_{t})\log(1-s_{t})\bigg{)} \tag{9}\]

where \(s_{t}\) is the score of the candidate triple \(t\), \(y_{t}=1\) for gold triples and \(0\) for others. We also train the entity tagger with a cross-entropy loss:

\[\mathcal{L}_{e}=-\frac{1}{n}\sum_{i=1}^{n}\sum_{*\in\{s,e\}}\log p(y_{i}^{*}= \hat{y}_{i}^{*}) \tag{10}\]

where \(\hat{y}_{i}^{s/e}\) are the gold start and end tags of the \(i\)-th word, respectively. Finally, we train the R-BPtrNet with the joint loss \(\mathcal{L}=\mathcal{L}_{t}+\mathcal{L}_{e}\).

To prevent error propagation, we use the gold entity tags to filter out valid candidate triples and compute the tag embeddings \(\mathbf{e}_{1:n}\) during training. We also update the memory \({\bf M}\) with the gold relational triples. During inference, we extract triples from scratch and use the predicted entity tags and relational triples instead of the gold ones.

## 4 Experiments

### Datasets and Evaluation Metrics

We evaluate our method on two benchmark datasets.

**NYT**Riedel et al. (2010) consists of sentences from the New York Times corpus and contains 24 relation types. **WebNLG**Gardent et al. (2017) was created for natural language generation task. It contains 171 relation types2 and was adopted for relational triple extraction by Zeng et al. (2018). We split the sentences into three categories: _Normal_, _SingleEntityOverlap_ (_SPO_) and _EntityPairOverlap_ (_EPO_) following Zeng et al. (2018). The statistics of the two datasets are shown in Table 1. Following previous work Zeng et al. (2018); Wei et al. (2020); Wang et al. (2020), an extracted relational triple is regarded as correct only if the relation and the heads of both subject and object are all correct. We report the standard micro precision, recall, and \(F_{1}\)-scores on both datasets.

Footnote 2: As mentioned in Wang et al. (2020), this number is miswritten as 246 in Wei et al. (2020) and Yu et al. (2019). Here we quote the correct number from Wang et al. (2020).

### Experimental Settings

We determine the hyper-parameters on the validation sets. We use the pre-trained GloVe Pennington et al. (2014) embeddings as \(\mathbf{w}\). We adopt a one-layer CNN with \(d_{c}=30\) channels to learn \(\mathbf{c}\) from 30-dimensional randomly-initialized character embeddings. We choose the state-of-the-art RoBERTaLMAGE Liu et al. (2019) model3 as the pre-tained LM. For a fair comparison with previous methods, we also conduct experiments and report the scores with BERTBASEDevlin et al. (2019). We set \(d_{in}\) (Equation 1) as 300. The hidden dimensions of the encoder \(d_{E}\) and the entity tagger \(d_{T}\) are both \(200\). The dimensions of entity tag embeddings \(d_{e}\) and relation type embeddings \(d_{R}\) are set as \(50\) and \(200\), respectively. The projection dimension \(d_{P}\) of the RN is set as \(500\).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Dataset} & \multicolumn{2}{c}{NYT} & \multicolumn{2}{c}{WebNLG} \\ \cline{2-5}  & Train & Test & Train & Test \\ \hline _Normal_ & 37013 & 3266 & 1596 & 246 \\ _SEO_ & 9782 & 1297 & 227 & 457 \\ _EPO_ & 14735 & 978 & 3406 & 26 \\ \hline ALL & 56195 & 5000 & 5019 & 703 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of evaluation datasets.

 

We add 10% dropout Srivastava et al. (2014) on the input of all LSTMs for regularization. Following previous work Zeng et al. (2018); Wei et al. (2020); Wang et al. (2020), we set the max length of input sentences to 100. We use the Adam optimizer Kingma and Ba (2014) to fine-tune the LM and train other parameters with the learning rates of \(10^{-5}\) and \(10^{-3}\), respectively. We train our model for 30/90 epochs with the batch size as 32/8 on NYT/WebNLG. At the beginning of the last 10 epochs, we load the parameters with the best validation performance and divide the learning rates by ten. Finally, we choose the best model on the validation set and output results on the test set.

### Performance Evaluation

We present our results on the NYT and WebNLG test sets in Table 2 and compare them with several previous state-of-the-art models:

* **NovelTagging**Zheng et al. (2017) transformed this task into a sequence tagging problem but neglected the overlapping triples.
* **CopyRE**Zeng et al. (2018) proposed a seq2seq model based on the copy mechanism to generate triple element as sequences.
* **CopyRE\({}_{RL}\)Zeng et al. (2019) proposed to learn the extraction order of CopyRE with Reinforcement Learning (RL).
* **GraphRel**Fu et al. (2019) proposed a graph convolutional network for this task.
* **ETL-Span**Yu et al. (2019) proposed a decomposition-based tagging scheme.
* **CopyMTL**Zeng et al. (2020) proposed a Multi-Task Learning (MTL) framework based on CopyRE to address multi-token entities.
* **WDec**Nayak and Ng (2020) proposed an encoder-decoder architecture for this task.
* **CGT\({}_{UnLM}\)Ye et al. (2020) proposed a generative transformer module with a triple contrastive training object.
* **CasRel**Wei et al. (2020) proposed a cascade binary tagging framework.
* **TPLinker**Wang et al. (2020) proposed a one-stage token pair linking model with a novel handshaking tagging scheme.

From Table 2 we have the following observations: (1) The R-BPtrNet significantly outperforms all previous non-LM methods. It demonstrates the superiority of our seq2seq-based framework to jointly extract explicit and implicit relational triples and improve the performance for this task. Additionally, the R-BPtrNet produces competitive performance to the BERT-based baseline models

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Method & \multicolumn{3}{c}{NYT} & \multicolumn{3}{c}{WebNLG} \\ \cline{2-6}  & Prec. & Rec. & \(F_{1}\) & Prec. & Rec. & \(F_{1}\) \\ \hline NovelTagging Zheng et al. (2017) & 62.4 & 31.7 & 42.0 & 52.5 & 19.3 & 28.3 \\ CopyRE Zeng et al. (2018) & 72.8 & 69.4 & 71.1 & 60.9 & 61.1 & 61.0 \\ CopyRE\({}_{RL}\)Zeng et al. (2019) & 77.9 & 67.2 & 72.1 & 63.3 & 59.9 & 61.6 \\ GraphRel Fu et al. (2019) & 63.9 & 60.0 & 61.9 & 44.7 & 41.1 & 42.9 \\ ETL-Span Yu et al. (2019) & 84.9 & 72.3 & 78.1 & 84.0 & 91.5 & 87.6 \\ CopyMTL Zeng et al. (2020) & 75.7 & 68.7 & 72.0 & 58.0 & 54.9 & 56.4 \\ WDec Nayak and Ng (2020) & 94.5 & 76.2 & 84.4 & - & - & - \\ CGT\({}_{UnILM}\)Ye et al. (2020) & **94.7** & 84.2 & 89.1 & 92.9 & 75.6 & 83.4 \\ CasRel\({}_{LSTM}\)Wei et al. (2020) & 84.2 & 83.0 & 83.6 & 86.9 & 80.6 & 83.7 \\ CasRel\({}_{BERT}\)Wei et al. (2020) & 89.7 & 89.5 & 89.6 & 93.4 & 90.1 & 91.7 \\ TPLinker\({}_{LSTM}\)Wang et al. (2020) & 83.8 & 83.4 & 83.6 & 90.8 & 90.3 & 90.5 \\ TPLinker\({}_{BERT}\)Wang et al. (2020) & 91.3 & 92.5 & 91.9 & 91.8 & 92.0 & 91.9 \\ \hline R-BPtrNet & 90.9 & 91.3 & 91.1 & 90.7 & **94.6** & 92.6 \\ R-BPtrNet\({}_{BERT}\) & 92.7 & 92.5 & 92.6 & 93.7 & 92.8 & 93.3 \\ R-BPtrNet\({}_{RoBERia}\) & 94.0 & **92.9** & **93.5** & **94.3** & 93.3 & **93.8** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of our method and previous state-of-the-art models on the NYT and WebNLG test sets. The best scores are in bold and the second-best scores are underlined. R-BPtrNet is our model without pre-trained LMs. R-BPtrNet\({}_{BERT}\) and R-BPtrNet\({}_{RoBERia}\) are models using BERTBASE and RoBERTaLARGE, respectively.

 

without using BERT. It shows that the improvements of our model come not primarily from the pre-trained LM representations, but from the introduction of relational reasoning patterns to this task. (2) R-BPtrNet\({}_{BERT}\) outperforms BERT-based baseline models. It indicates that our method can effectively extract implicit relational triples with the assistance of the triple-retaining external memory and the pattern-capturing RN. (3) R-BPtrNet\({}_{RoBERTa}\) further outperforms R-BPtrNet\({}_{BERT}\) and other baseline methods. It indicates that the more powerful LM brings more prior knowledge and real-world relational facts, enhancing the model's ability to learn real-world relational reasoning patterns.

### Performance on Different Sentence Types

To demonstrate the ability of our model in handling the multiple triples and overlapping triples of a sentence, we split the test sets of NYT and WebNLG datasets according to the overlapping patterns and the number of triples. We conduct further experiments on these subsets and report the results in Table 3, from which we can observe that: (1) The R-BPtrNet\({}_{RoBERTa}\) and R-BPtrNet\({}_{BERT}\) both significantly outperform previous models on the SPO and EPO subsets of NYT and WebNLG datasets. It proves the validity of our method to address the overlapping triple problem. Moreover, we find that implicit relational triples usually overlap with others. Therefore, the improvements on the overlapping subsets also validate the effectiveness of our method for extracting implicit relational triples. (2) R-BPtrNet\({}_{RoBERTa}\) and R-BPtrNet\({}_{BERT}\) both bring improvements to sentences with multiple triples compared to baseline models. It indicates that our method can effectively extract multiple relational triples from a sentence. Furthermore, we observe more significant improvements when the number of triples grows. We hypothesize that this is because implicit relational triples are more likely to occur in sentences with more triples. Our model extracts the implicit relational triples more correctly and improves the performance.

### Ablation Study on Implicit Triples

We run an ablation study to investigate the contribution of each component in our model to the implicit relational triples. We manually select 134 sentences with rich implicit triples from the NYT test set4. We conduct experiments on the subset

Figure 3: An ablation study on a manually selected subset with rich implicit relational triples.

 

using the following ablation options:

* **R-BPtrNet\({}_{RoBERTa}\)** and **R-BPtrNet\({}_{BERT}\)** are the full models using RoBERTaLARGE and BERTBASE as LMs, respectively.
* **R-BPtrNet** removes the **pre-trained LM representations** from the full model.
* **BPtrNet** removes **the RN** from the R-BPtrNet. Under this setting, we feed a gated summation of the memory into the decoder's input of the next time step.
* **BPtrNet\({}_{NoMem}\)** removes **the external memory** from the BPtrNet, which means that the previously extracted triples are not retained.

We compare the performance of these options with the previous BERT-based models. We also analyze the performance on predicting only the entity pairs and the relations, respectively. We illustrate the results in Figure 3, from which we can observe that: (1) BPtrNet\({}_{NoMem}\) produces comparable results to the baseline models. We speculate that it benefits from the seq2seq structure and the previous triples are embedded into the decoder's hidden states. (2) BPtrNet brings huge improvements over the BPtrNet\({}_{NoMem}\) to the entity pair and the triple \(F_{1}\) scores. It indicates that the external memory effectively helps discover entity pairs that have implicit relational connections by retaining previously extracted triples. (3) R-BPtrNet brings significant improvements over the BPtrNet to the relation and the triple \(F_{1}\) scores. It indicates that the RN effectively captures the relational reasoning patterns and enhances the relation type inference of implicit relations. (4) The pre-trained LMs only bring minor improvements. It proves that the effectiveness of our model comes primarily from the external memory and the introduction of relational reasoning patterns rather than the pre-trained LMs.

### Case Study

Figure 4 shows the comparison of the best previous model TPLinker\({}_{BERT}\) and our R-BPtrNet\({}_{BERT}\) model on three example sentences from the implicit subset in Section 4.5. The first example contains the transitive pattern of the relation "_contains_". The second example contains a multi-hop relation path pattern between "_Chad Hurley_" and "_Google_" through the intermediate entity "_Youtube_". The third example contains a composite pattern between the siblings "_Crowley_" and "_Edwin Edwards_" with a common ancestor "_Edmund_". We can observe that the TPLinker\({}_{BERT}\) model fails to extract the implicit relational triples. The R-BPtrNet\({}_{BERT}\) successfully captures various reasoning patterns in the real world and effectively extracts all the implicit relational triples in the examples.

Figure 4: Examples of sentences with implicit relational triples, and the predictions from the TPLinker\({}_{BERT}\) and R-BPtrNet\({}_{BERT}\) model. Bold and colored texts are entities. We distinguish different entities with different colors. Explicit and implicit relational triples are represented by black and green solid arrows, respectively. Blue dashed arrows indicate explicit relational connections between entities, but they do not appear as relational triples because their relation types don’t belong to the pre-defined relations of the dataset.

 

Conclusion

In this paper, we propose a unified framework to extract explicit and implicit relational triples jointly. To discover entity pairs that may have implicit relational connections, we propose a binary pointer network to extract relational triples relevant to each word sequentially and introduce an external memory to retain the extracted triples. To derive the relation types of the implicitly connected entity pairs, we propose to introduce real-world relational reasoning patterns to this task and capture the reasoning patterns with a relation network. We conduct experiments on two benchmark datasets, and the results prove the effectiveness of our method.

## Acknowledgement

We would like to thank the anonymous reviewers for their constructive comments on this paper. This work was supported by the National Natural Science Foundation of China under Grant numbers U1936208, U1936216, and 61862002.

## References

* Angeli and Manning (2013) Gabor Angeli and Christopher D Manning. 2013. Philosophers are mortal: Inferring the truth of unseen facts. In CoNLL, pp. 133-142. Cited by: SS1.
* Bahdanau et al. (2015) Neural machine translation by jointly learning to align and translate. In ICLR, Cited by: SS1.
* S. Chan and D. Roth (2011) Exploiting syntactico-semantic structures for relation extraction. In ACL-HLT, pp. 551-560. Cited by: SS1.
* Dai et al. (2019) Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In AAAI, Vol. 33, pp. 6300-6308. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, Cited by: SS1.
* X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang (2014) Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In KDD, pp. 601-610. Cited by: SS1.
* T. Fu, P. Li, and W. Ma (2019) Graphrel: modeling text as relational graphs for joint entity and relation extraction. In ACL, pp. 1409-1418. Cited by: SS1.
* C. Gardent, A. Shimorina, B. Narayan, and L. Perez-Beltrachini (2017) Creating training corpora for nlg micro-planning. In ACL, Cited by: SS1.
* F. A. Gers, J. Schmidhuber, and F. Cummins (2000) Learning to forget: continual prediction with lstm. Neural computation12 (10), pp. 2451-2471. Cited by: SS1.
* Matthew R Gormley, Mo Yu, and Mark Dredze (2015) Improved relation extraction with feature-rich compositional embedding models. In EMNLP, pp. 1774-1784. Cited by: SS1.
* P. Gupta, H. Schutze, and B. Andrassy (2016) Table filling multi-task recurrent neural network for joint entity and relation extraction. In COLING, pp. 2537-2547. Cited by: SS1.
* N. Jia, X. Cheng, and Sen Su (2020) Improving new-proving knowledge graph embedding using locally and globally attentive relation paths. In European Conference on Information Retrieval, pp. 17-32. Cited by: SS1.
* K. Jun, C. Yixin, Z. Jianbing, H. Xiangnan, G. Ming, and Z. Aoying (2020) Improving neural relation extraction with implicit mutual relations. In ICDE, pp. 1021-1032. Cited by: SS1.
* D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS1.
* Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel (1989) Backpropagation applied to handwritten zip code recognition. Neural computation1 (4), pp. 541-551. Cited by: SS1.
* Q. Li and H. Ji (2014) Incremental joint extraction of entity mentions and relations. In ACL, pp. 402-412. Cited by: SS1.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1.
* M. Miwa and M. Bansal (2016) End-to-end relation extraction using lstms on sequences and tree structures. In ACL, pp. 1105-1116. Cited by: SS1.
* Tapas Nayak and H. Tou Ng (2020) Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In AAAI, Vol. 34, pp. 8528-8535. Cited by: SS1.
* Jeffrey Pennington, Richard Socher, and C. D. Manning (2014) Glove: global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543.

 

Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In _WWW_, pages 1015-1024.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_, pages 148-163. Springer.
* Santoro et al. (2017) Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In _Advances in neural information processing systems_, pages 4967-4976.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The journal of machine learning research_, 15(1):1929-1958.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Advances in neural information processing systems_, pages 2692-2700.
* Wang et al. (2018) Shaolei Wang, Yue Zhang, Wanxiang Che, and Ting Liu. 2018. Joint extraction of entities and relations based on a novel graph scheme. In _IJCAI_, pages 4461-4467.
* Wang et al. (2020) Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking. _arXiv preprint arXiv:2010.13415_.
* Wei et al. (2020) Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In _ACL_, pages 1476-1488.
* Ye et al. (2020) Hongbin Ye, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang, and Huajun Chen. 2020. Contrastive triple extraction with generative transformer. _arXiv preprint arXiv:2009.06207_.
* Yu et al. (2019) Bowen Yu, Zhenyu Zhang, and Jianlin Su. 2019. Joint extraction of entities and relations based on a novel decomposition strategy. _arXiv preprint arXiv:1909.04273_.
* Yu and Lam (2010) Xiaofeng Yu and Wai Lam. 2010. Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. In _COLING 2010: Posters_, pages 1399-1407.
* Zelenko et al. (2003) Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. _Journal of machine learning research_, 3(Feb):1083-1106.
* Zeng et al. (2020) Daojian Zeng, Haoran Zhang, and Qianying Liu. 2020. Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning. In _AAAI_, pages 9507-9514.
* Zeng et al. (2019) Xiangrong Zeng, Shizhu He, Daiqian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In _EMNLP-IJCNLP_, pages 367-377.
* Zeng et al. (2018) Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In _ACL_, pages 506-514.
* Zheng et al. (2017) Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In _ACL_, pages 1227-1236.
* Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In _ACL_, pages 427-434.
* Zhu et al. (2019) Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019. Graph neural networks with generated parameters for relation extraction. In _ACL_, pages 1331-1339