SlowGuess/NAACL_2021_OCR huggingface dataset:
We add  $10\%$  dropout (Srivastava et al., 2014) on the input of all LSTMs for regularization. Following previous work (Zeng et al., 2018; Wei et al., 2020; Wang et al., 2020), we set the max length of input sentences to 100. We use the Adam optimizer (Kingma and Ba, 2014) to fine-tune the LM and train other parameters with the learning rates of  $10^{-5}$  and  $10^{-3}$ , respectively. We train our model for 30/90 epochs with the batch size as 32/8 on NYT/WebNLG. At the beginning of the last 10 epochs, we load the parameters with the best validation performance and divide the learning rates by ten. Finally, we choose the best model on the validation set and output results on the test set.
We present our results on the NYT and WebNLG test sets in Table 2 and compare them with several previous state-of-the-art models:
- NovelTagging (Zheng et al., 2017) transformed this task into a sequence tagging problem but neglected the overlapping triples.
- CopyRE (Zeng et al., 2018) proposed a seq2seq model based on the copy mechanism to generate triple element as sequences.
- CopyRE $_{RL}$  (Zeng et al., 2019) proposed to learn the extraction order of CopyRE with
Reinforcement Learning (RL).
- GraphRel (Fu et al., 2019) proposed a graph convolutional network for this task.
ETL-Span (Yu et al., 2019) proposed a decomposition-based tagging scheme.
- CopyMTL (Zeng et al., 2020) proposed a Multi-Task Learning (MTL) framework based on CopyRE to address multi-token entities.
- WDec (Nayak and Ng, 2020) proposed an encoder-decoder architecture for this task.
-  $\mathbf{CGT}_{\mathit{UniLM}}$  (Ye et al., 2020) proposed a generative transformer module with a triple contrastive training object.
- CASREL (Wei et al., 2020) proposed a cascade binary tagging framework.
- TPLinker (Wang et al., 2020) proposed a one-stage token pair linking model with a novel handshaking tagging scheme.
From Table 2 we have the following observations: (1) The R-BPtrNet significantly outperforms all previous non-LM methods. It demonstrates the superiority of our seq2seq-based framework to jointly extract explicit and implicit relational triples and improve the performance for this task. Additionally, the R-BPtrNet produces competitive performance to the BERT-based baseline models
Method	NYT	WebNLG
Nor.	SEO	EPO	N=1	N=2	N=3	N=4	N≥5	Nor.	SEO	EPO	N=1	N=2	N=3	N=4	N≥5
CopyRE	66.0	48.6	55.0	67.1	58.6	52.0	53.6	30.0	59.2	33.0	36.6	59.2	42.5	31.7	24.2	30.0
\( CopyRE_{RL} \)	71.2	69.4	72.8	71.7	72.6	72.5	77.9	45.9	65.4	60.1	67.4	63.4	62.2	64.4	57.2	55.7
GraphRel	69.6	51.2	58.2	71.0	61.5	57.4	55.1	41.1	65.8	38.3	40.6	66.0	48.3	37.0	32.1	32.1
\( ETL-Span^{\dagger} \)	88.5	87.6	60.3	85.5	82.1	74.7	75.6	76.9	87.3	91.5	80.5	82.1	86.5	91.4	89.5	91.1
\( CASREL_{BERT} \)	87.3	91.4	92.0	88.2	90.3	91.9	94.2	83.7	89.4	92.2	94.7	89.3	90.8	94.2	92.4	90.9
\( TPLinker_{BERT} \)	90.1	93.4	94.0	90.0	92.9	93.1	96.1	90.0	87.9	92.5	95.3	88.0	90.1	94.6	93.3	91.6
R-BPtrNet \( _{BERT} \)	90.4	94.4	95.2	89.5	93.1	93.5	96.7	91.3	89.5	93.9	96.1	88.5	91.4	96.2	94.9	94.2
R-BPtrNet \( _{ROBERTa} \)	91.2	95.3	96.1	90.5	93.6	94.2	97.7	92.1	89.9	94.4	97.4	89.3	91.7	96.5	95.8	94.8
Table 3:  $F_{1}$  scores on sentences with different overlapping patterns and different triple numbers. The best scores are in bold and the second-best scores are underlined.  $\dagger$  marks scores reproduced by (Wang et al., 2020).
without using BERT. It shows that the improvements of our model come not primarily from the pre-trained LM representations, but from the introduction of relational reasoning patterns to this task. (2) R-BPtrNet $_{BERT}$  outperforms BERT-based baseline models. It indicates that our method can effectively extract implicit relational triples with the assistance of the triple-retaining external memory and the pattern-capturing RN. (3) R-BPtrNet $_{ROBERTa}$  further outperforms R-BPtrNet $_{BERT}$  and other baseline methods. It indicates that the more powerful LM brings more prior knowledge and real-world relational facts, enhancing the model's ability to learn real-world relational reasoning patterns.
4.4 Performance on Different Sentence Types
To demonstrate the ability of our model in handling the multiple triples and overlapping triples of a sentence, we split the test sets of NYT and WebNLG datasets according to the overlapping
Figure 3: An ablation study on a manually selected subset with rich implicit relational triples.
patterns and the number of triples. We conduct further experiments on these subsets and report the results in Table 3, from which we can observe that: (1) The R-BPtrNet $_{R_{O}BERTa}$  and R-BPtrNet $_{BERT}$  both significantly outperform previous models on the SPO and EPO subsets of NYT and WebNLG datasets. It proves the validity of our method to address the overlapping triple problem. Moreover, we find that implicit relational triples usually overlap with others. Therefore, the improvements on the overlapping subsets also validate the effectiveness of our method for extracting implicit relational triples. (2) R-BPtrNet $_{R_{O}BERTa}$  and R-BPtrNet $_{BERT}$  both bring improvements to sentences with multiple triples compared to baseline models. It indicates that our method can effectively extract multiple relational triples from a sentence. Furthermore, we observe more significant improvements when the number of triples grows. We hypothesize that this is because implicit relational triples are more likely to occur in sentences with more triples. Our model extracts the implicit relational triples more correctly and improves the performance.
4.5 Ablation Study on Implicit Triples
We run an ablation study to investigate the contribution of each component in our model to the implicit relational triples. We manually select 134 sentences with rich implicit triples from the NYT test set<sup>4</sup>. We conduct experiments on the subset
4We first select sentences that contain at least two overlapping relational triples. For example, if a sentence contains entity  $A$ ,  $B$  and  $C$ , and if  $A \rightarrow B$  and  $B \rightarrow C$  exists or  $A \rightarrow B$  and  $A \rightarrow C$  exists, this sentence is selected. Note that  $A \rightarrow B$  and  $B \rightarrow A$  are counted as one triple during this selecting procedure. Then, we manually check all selected sentences and keep the ones with implicit relational triples.
Sentence	... organizing an expedition starting in November in Jinghong, a small city in the Yunnan province in China.	We're providing a new outlet for them for distribution," said Chad Hurley, chief executive and ... of YouTube, a division of Google.	... Edmund, was an influential municipal judge in Crowley, who was ... as well as a close adviser to former Louisiana Gov. Edwin Edwards ...
Ground Truth	Yunnan
Jinghong
China	YouTube
Chad
Hurley
Google	Edmund
| lives in
| Crowley |
| Luisiana
TPLinker	Yunnan
Jinghong
China	YouTube
company
| Chad
Hurley
Google	Edmund
| lives in
| Crowley |
| Luisiana
Ours	Yunnan
Jinghong
China	YouTube
company
| Chad
Hurley
Google	Edmund---advisor→Edwin
| lives in
| Crowley |
| Luisiana
Figure 4: Examples of sentences with implicit relational triples, and the predictions from the TPLinker  $BERT$  and R-BPtrNet  $BERT$  model. Bold and colored texts are entities. We distinguish different entities with different colors. Explicit and implicit relational triples are represented by black and green solid arrows, respectively. Blue dashed arrows indicate explicit relational connections between entities, but they do not appear as relational triples because their relation types don't belong to the pre-defined relations of the dataset.
using the following ablation options:
- R-BPtrNet  $R_{O B E R T a}$  and R-BPtrNet  $B E R T$  are the full models using RoBERTa  $\mathrm{LARGE}$  and BERT  $\mathrm{BASE}$  as LMs, respectively.
- R-BPtrNet removes the pre-trained LM representations from the full model.
- BPtrNet removes the RN from the R-BPtrNet. Under this setting, we feed a gated summation of the memory into the decoder's input of the next time step.
- BPtrNetNoMem removes the external memory from the BPtrNet, which means that the previously extracted triples are not retained.
We compare the performance of these options with the previous BERT-based models. We also analyze the performance on predicting only the entity pairs and the relations, respectively. We illustrate the results in Figure 3, from which we can observe that: (1)  $\mathrm{BPtrNet}_{\mathrm{NoMem}}$  produces comparable results to the baseline models. We speculate that it benefits from the seq2seq structure and the previous triples are embedded into the decoder's hidden states. (2) BPtrNet brings huge improvements over the  $\mathrm{BPtrNet}_{\mathrm{NoMem}}$  to the entity pair and the triple  $F_{1}$  scores. It indicates that the external memory effectively helps discover entity pairs that have implicit relational connections by retaining previously
extracted triples. (3) R-BPtrNet brings significant improvements over the BPtrNet to the relation and the triple  $F_{1}$  scores. It indicates that the RN effectively captures the relational reasoning patterns and enhances the relation type inference of implicit relations. (4) The pre-trained LMs only bring minor improvements. It proves that the effectiveness of our model comes primarily from the external memory and the introduction of relational reasoning patterns rather than the pre-trained LMs.
Figure 4 shows the comparison of the best previous model TPLinker  $BERT$  and our R-BPtrNet  $BERT$  model on three example sentences from the implicit subset in Section 4.5. The first example contains the transitive pattern of the relation "contains". The second example contains a multi-hop relation path pattern between "Chad Hurley" and "Google" through the intermediate entity "Youtube". The third example contains a composite pattern between the siblings "Crowley" and "Edwin Edwards" with a common ancestor "Edmund". We can observe that the TPLinker  $BERT$  model fails to extract the implicit relational triples. The R-BPtrNet  $BERT$  successfully captures various reasoning patterns in the real world and effectively extracts all the implicit relational triples in the examples.
In this paper, we propose a unified framework to extract explicit and implicit relational triples jointly. To discover entity pairs that may have implicit relational connections, we propose a binary pointer network to extract relational triples relevant to each word sequentially and introduce an external memory to retain the extracted triples. To derive the relation types of the implicitly connected entity pairs, we propose to introduce real-world relational reasoning patterns to this task and capture the reasoning patterns with a relation network. We conduct experiments on two benchmark datasets, and the results prove the effectiveness of our method.
We would like to thank the anonymous reviewers for their constructive comments on this paper. This work was supported by the National Natural Science Foundation of China under Grant numbers U1936208, U1936216, and 61862002.
Gabor Angeli and Christopher D Manning. 2013. Philosophers are mortal: Inferring the truth of unseen facts. In CoNLL, pages 133-142.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In ACL-HLT, pages 551-560.
Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiao-qiao She, and Haifeng Wang. 2019. Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In AAAI, volume 33, pages 6300–6308.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In KDD, pages 601-610.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In ACL, pages 1409-1418.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planning. In ACL.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12(10):2451-2471.
Matthew R Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. In EMNLP, pages 1774-1784.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In COLING, pages 2537-2547.
Ningning Jia, Xiang Cheng, and Sen Su. 2020. Improving knowledge graph embedding using locally and globally attentive relation paths. In European Conference on Information Retrieval, pages 17-32. Springer.
Kuang Jun, Cao Yixin, Zheng Jianbing, He Xiangnan, Gao Ming, and Zhou Aoying. 2020. Improving neural relation extraction with implicit mutual relations. In ICDE, pages 1021-1032.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Yann LeCun, Bernhard Boser, John S Denker, Bonnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. *Neural computation*, 1(4):541-551.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In ACL, pages 402-412.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In ACL, pages 1105-1116.
Tapas Nayak and Hwee Tou Ng. 2020. Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In AAAI, volume 34, pages 8528-8535.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In WWW, pages 1015-1024.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 148-163. Springer.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in neural information processing systems, pages 2692-2700.
Shaolei Wang, Yue Zhang, Wanxiang Che, and Ting Liu. 2018. Joint extraction of entities and relations based on a novel graph scheme. In *IJCAI*, pages 4461-4467.
Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking. arXiv preprint arXiv:2010.13415.
Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In ACL, pages 1476-1488.
Hongbin Ye, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang, and Huajun Chen.
2020. Contrastive triple extraction with generative transformer. arXiv preprint arXiv:2009.06207.
Bowen Yu, Zhenyu Zhang, and Jianlin Su. 2019. Joint extraction of entities and relations based on a novel decomposition strategy. arXiv preprint arXiv:1909.04273.
Xiaofeng Yu and Wai Lam. 2010. Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. In *COLING* 2010: Posters, pages 1399-1407.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of machine learning research, 3(Feb):1083-1106.
Daojian Zeng, Haoran Zhang, and Qianying Liu. 2020. Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning. In AAAI, pages 9507-9514.
Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In EMNLP-IJCNLP, pages 367-377.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In ACL, pages 506-514.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In ACL, pages 1227-1236.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In ACL, pages 427-434.
Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019. Graph neural networks with generated parameters for relation extraction. In ACL, pages 1331-1339.