SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {j = 1} ^ {| D |} \left[ \sum_ {s \in T _ {j}} \log p _ {\theta} (s | \mathbf {x} _ {j}) + \sum_ {r \in T _ {j} | s} \log p _ {\phi_ {r}} (o | s, \mathbf {x} _ {j}) \right. \\ \left. + \sum_ {r \in R \backslash T _ {j} | s} \log p _ {\phi_ {r}} \left(o _ {\varnothing} \mid s, \mathbf {x} _ {j}\right) \right]. \tag {12} \\ \end{array}
$$
where parameters  $\Theta = \{\theta, \{\phi_r\}_{r \in R}\}$ .  $p_{\theta}(s|\mathbf{x})$  is defined in Eq. (8) and  $p_{\phi_r}(o|s,\mathbf{x})$  is defined in Eq. (11). We train the model by maximizing  $J(\Theta)$  through Adam stochastic gradient descent (Kingma and Ba, 2014) over shuffled mini-batches.
Datasets and Evaluation Metrics We evaluate the framework on two public datasets NYT (Riedel
Category	NYT	WebNLG
Train	Test	Train	Test
Normal	37013	3266	1596	246
EPO	9782	978	227	26
SEO	14735	1297	3406	457
ALL	56195	5000	5019	703
Table 1: Statistics of datasets. Note that a sentence can belong to both EPO class and SEO class.
et al., 2010) and WebNLG (Gardent et al., 2017). NYT dataset was originally produced by distant supervision method. It consists of 1.18M sentences with 24 predefined relation types. WebNLG dataset was originally created for Natural Language Generation (NLG) tasks and adapted by (Zeng et al., 2018) for relational triple extraction task. It contains 246 predefined relation types. The sentences in both datasets commonly contain multiple relational triples, thus NYT and WebNLG datasets suit very well to be the testbed for evaluating model on extracting overlapping relational triples<sup>4</sup>. We use the datasets released by (Zeng et al., 2018), in which NYT contains 56195 sentences for training, 5000 sentences for validation, and 5000 sentences for test, and WebNLG contains 5019 sentences for training, 500 sentences for validation and 703 sentences for test. According to different overlapping patterns of relational triples, we split the sentences into three categories, namely, Normal, EntityPairOverlap (EPO) and SingleEntityOverlap (SEO) for detailed experiments on different types of overlapping relational triples. The statistics of the two datasets are described in Table 1.
Following previous work (Fu et al., 2019), an extracted relational triple (subject, relation, object) is regarded as correct only if the relation and the heads of both subject and object are all correct. For fair comparison, we report the standard micro Precision (Prec.), Recall (Rec.) and F1-score as in line with baselines.
Implementation Details The hyper-parameters are determined on the validation set. More implementation details are described in Appendix A.
${}^{4}$  Datasets such as ACE, Wiki-KBP have few overlapping triples in the sentences hence are not suitable for evaluating the performance of overlapping triple extraction. Nonetheless, to validate the generality of the proposed framework, we also conduct supplemental experiments on these datasets along with the comparison of 12 recent strong baselines. The results of the comprehensive comparison, which show consistent superiority of our model over most compared methods, can be found in Appendix C.
Method	NYT	WebNLG
Prec.	Rec.	F1	Prec.	Rec.	F1
NovelTagging (Zheng et al., 2017)	62.4	31.7	42.0	52.5	19.3	28.3
CopyROneDecoder (Zeng et al., 2018)	59.4	53.1	56.0	32.2	28.9	30.5
CopyRMultiDecoder (Zeng et al., 2018)	61.0	56.6	58.7	37.7	36.4	37.1
GraphRel1p (Fu et al., 2019)	62.9	57.3	60.0	42.3	39.2	40.7
GraphRel2p (Fu et al., 2019)	63.9	60.0	61.9	44.7	41.1	42.9
CopyRRL (Zeng et al., 2019)	77.9	67.2	72.1	63.3	59.9	61.6
CopyR*RL	72.8	69.4	71.1	60.9	61.1	61.0
CASRELrandom	81.5	75.7	78.5	84.7	79.5	82.0
CASRELLSTM	84.2	83.0	83.6	86.9	80.6	83.7
CASREL	89.7	89.5	89.6	93.4	90.1	91.8
Table 2: Results of different methods on NYT and WebNLG datasets. Our re-implementation is marked by * .
4.2 Experimental Result
Compared Methods We compare our model with several strong state-of-the-art models, namely, NovelTagging (Zheng et al., 2017), CopyR (Zeng et al., 2018), GraphRel (Fu et al., 2019) and CopyR $_{RL}$  (Zeng et al., 2019). The reported results for the above baselines are directly copied from the original published literature.
Note that we instantiate the CASREL framework on top of a pre-trained BERT model to combine the power of the proposed novel tagging scheme and the pre-learned prior knowledge for better performance. To evaluate the impact of introducing the Transformer-based BERT model, we conduct a set of ablation tests. CASRELrandom is the framework where all parameters of BERT are randomly initialized; CASREL LSTM is the framework instantiated on a LSTM-based structure as in (Zheng et al., 2017) with pre-trained Glove embedding (Pennington et al., 2014); CASREL is the full-fledged framework using pre-trained BERT weights.
Main Results Table 2 shows the results of different baselines for relational triple extraction on two datasets. The CASREL model overwhelmingly outperforms all the baselines in terms of all three evaluation metrics and achieves encouraging  $17.5\%$  and  $30.2\%$  improvements in F1-score over the best state-of-the-art method (Zeng et al., 2019) on NYT and WebNLG datasets respectively. Even without taking advantage of the pre-trained BERT, CASREL random and CASREL LSTM are still competitive to existing state-of-the-art models. This validates the utility of the proposed cascade decoder that adopts a novel binary tagging scheme. The performance improvements from CASREL random to CASREL highlight the importance of the prior knowledge in a pre-trained language model.
We can also observe from the table that there is a significant gap between the performance on NYT and WebNLG datasets for existing models, and we believe this gap is due to their drawbacks in dealing with overlapping triples. More precisely, as presented in Table 1, we can find that NYT dataset is mainly comprised of Normal class sentences while the majority of sentences in WebNLG dataset belong to EPO and SEO classes. Such inconsistent data distribution of two datasets leads to a comparatively better performance on NYT and a worse performance on WebNLG for all the baselines, exposing their drawbacks in extracting overlapping relational triples. In contrast, the CASREL model and its variants (i.e., CASRELrandom and CASREL LSTM) all achieve a stable and competitive performance on both NYT and WebNLG datasets, demonstrating the effectiveness of the proposed framework in solving the overlapping problem.
Detailed Results on Different Types of Sentences To further study the capability of the proposed CASREL framework in extracting overlapping relational triples, we conduct two extended experiments on different types of sentences and compare the performance with previous works.
The detailed results on three different overlapping patterns are presented in Figure 3. It can be seen that the performance of most baselines on Normal, EPO and SEO presents a decreasing trend, reflecting the increasing difficulty of extracting relational triples from sentences with different overlapping patterns. That is, among the three overlapping patterns, Normal class is the easiest pattern while EPO and SEO classes are the relatively harder ones for baseline models to extract. In contrast, the proposed CASREL model attains consistently strong performance over all three overlapping patterns, es
(a) F1 of Normal Class
(b) F1 of EntityPairOverlap Class
(c) F1 of SingleEntityOverlap Class
Figure 3: F1-score of extracting relational triples from sentences with different overlapping pattern.
Method	NYT	WebNLG
N=1	N=2	N=3	N=4	N≥5	N=1	N=2	N=3	N=4	N≥5
CopyROneDecoder	66.6	52.6	49.7	48.7	20.3	65.2	33.0	22.2	14.2	13.2
CopyRMultiDecoder	67.1	58.6	52.0	53.6	30.0	59.2	42.5	31.7	24.2	30.0
GraphRel1p	69.1	59.5	54.4	53.9	37.5	63.8	46.3	34.7	30.8	29.4
GraphRel2p	71.0	61.5	57.4	55.1	41.1	66.0	48.3	37.0	32.1	32.1
CopyR*RL	71.7	72.6	72.5	77.9	45.9	63.4	62.2	64.4	57.2	55.7
CASREL	88.2	90.3	91.9	94.2	83.7 (+37.8)	89.3	90.8	94.2	92.4	90.9 (+35.2)
Table 3: F1-score of extracting relational triples from sentences with different number (denoted as N) of triples.
ppecially for those hard patterns. We also validate the CASREL's capability in extracting relational triples from sentences with different number of triples. We split the sentences into five classes and Table 3 shows the results. Again, the CASREL model achieves excellent performance over all five classes. Though it's not surprising to find that the performance of most baselines decreases with the increasing number of relational triples that a sentence contains, some patterns still can be observed from the performance changes of different models. Compared to previous works that devote to solving the overlapping problem in relational triple extraction, our model suffers the least from the increasing complexity of the input sentence. Though the CASREL model gain considerable improvements on all five classes compared to the best state-of-the-art method CopyR $_{RL}$  (Zeng et al., 2019), the greatest improvement of F1-score on the two datasets both come from the most difficult class  $(N \geq 5)$ , indicating that our model is more suitable for complicated scenarios than the baselines.
Both of these experiments validate the superiority of the proposed cascade binary tagging framework in extracting multiple (possibly overlapping) relational triples from complicated sentences compared to existing methods. Previous works have to explicitly predict all possible relation types con
tained in a given sentence, which is quite a challenging task, and thus many relations are missing in their extracted results. In contrast, our CASREL model side-steps the prediction of relation types and tends to extract as many relational triples as possible from a given sentence. We attribute this to the relation-specific object tagger setting in high level tagging module of the cascade decoder that considers all the relation types simultaneously.
In this paper, we introduce a novel cascade binary tagging framework (CASREL) derived from a principled problem formulation for relational triple extraction. Instead of modeling relations as discrete labels of entity pairs, we model the relations as functions that map subjects to objects, which provides a fresh perspective to revisit the relational triple extraction task. As a consequent, our model can simultaneously extract multiple relational triples from sentences, without suffering from the overlapping problem. We conduct extensive experiments on two widely used datasets to validate the effectiveness of the proposed CASREL framework. Experimental results show that our model overwhelmingly outperforms state-of-the-art baselines over different scenarios, especially on the extraction of overlapping relational triples.
The authors would like to thank the anonymous referees for their valuable comments. This work is supported by the National Natural Science Foundation of China (No.61976102, No.U19A2065).
Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, pages 722-735.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250.
Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 551-560. Association for Computational Linguistics.
Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiao-qiao She, and Haifeng Wang. 2019. Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6300-6308.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 601-610.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1409-1418.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planners. In Proceedings
of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179-188.
Matthew R Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1774-1784.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2537-2547.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 541-550.
Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 917-928.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 402-412.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Annual Meeting of the Association for Computational Linguistics (ACL).
Liyuan Liu, Xiang Ren, Qi Zhu, Shi Zhi, Huan Gui, Heng Ji, and Jiawei Han. 2017. Heterogeneous supervision for relation extraction: A representation learning approach. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 46-56.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1003-1011.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree
structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1105-1116.
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1858-1869.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In Proceedings of the 26th International Conference on World Wide Web, pages 1015-1024.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pages 148-163.
Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. 2019. A hierarchical framework for relation extraction with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7072-7079.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Xiaofeng Yu and Wai Lam. 2010. Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1399-1407. Association for Computational Linguistics.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of machine learning research, 3(Feb):1083-1106.
Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In Proceedings
of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 367-377.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 506-514.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1227-1236.
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge-enriched transformer for emotion detection in textual conversations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 165-176.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL05), pages 427-434.
We adopt mini-batch mechanism to train our model with batch size as 6; the learning rate is set to  $1\mathrm{e}^{-5}$ ; the hyper-parameters are determined on the validation set. We also adopt early stopping mechanism to prevent the model from over-fitting. Specifically, we stop the training process when the performance on validation set does not gain any improvement for at least 7 consecutive epochs. The number of stacked bidirectional Transformer blocks  $N$  is 12 and the size of hidden state  $\mathbf{h}_N$  is 768. The pre-trained BERT model we used is [BERT-Base, Cased] $^5$ , which contains 110M parameters.
For fair comparison, the max length of input sentence to our model is set to 100 words as previous works (Zeng et al., 2018; Fu et al., 2019) suggest. We did not tune the threshold for both start and end position taggers to predict tag 1, but heuristically set the threshold to 0.5 as default. The performance might be better after carefully tuning the threshold, however it is beyond the research scope of this paper.
B Error Analysis
To explore the factors that affect the extracted relational triples of the CASREL model, we analyze the performance on predicting different elements of the triple  $(E1, R, E2)$  where  $E1$  represents the subject entity,  $E2$  represents the object entity and  $R$  represents the relation between them. An element like  $(E1, R)$  is regarded as correct only if the subject and the relation in the predicted triple  $(E1, R, E2)$  are both correct, regardless of the correctness of the predicted object. Similarly, we say an instance of  $E1$  is correct as long as the subject in the extracted triple is correct, so is  $E2$  and  $R$ .
Table 4 shows the results on different relational triple elements. For NYT, the performance on  $E1$  and  $E2$  are consistent with that on  $(E1, R)$  and  $(R, E2)$ , demonstrating the effectiveness of our proposed framework in identifying both subject and object entity mentions. We also find that there is only a trivial gap between the F1-score on  $(E1, E2)$  and  $(E1, R, E2)$ , but an obvious gap between  $(E1, R, E2)$  and  $(E1, R)/(R, E2)$ . It reveals that most relations for the entity pairs in extracted triples are correctly identified while some extracted entities
Element	NYT	WebNLG
Prec.	Rec.	F1	Prec.	Rec.	F1
E1	94.6	92.4	93.5	98.7	92.8	95.7
E2	94.1	93.0	93.5	97.7	93.0	95.3
R	96.0	93.8	94.9	96.6	91.5	94.0
(E1, R)	93.6	90.9	92.2	94.8	90.3	92.5
(R, E2)	93.1	91.3	92.2	95.4	91.1	93.2
(E1, E2)	89.2	90.1	89.7	95.3	91.7	93.5
(E1, R, E2)	89.7	89.5	89.6	93.4	90.1	91.8
Table 4: Results on relational triple elements.
fail to form a valid relational triple. In other words, it implies that identifying relations is somehow easier than identifying entities for our model.
In contrast to NYT, for WebNLG, the performance gap between  $(E1,E2)$  and  $(E1,R,E2)$  is comparatively larger than that between  $(E1,R,E2)$  and  $(E1,R)/(R,E2)$ . It shows that misidentifying the relations will bring more performance degradation than misidentifying the entities. Such observation also indicates that it's more challenging for the proposed CASREL model to identify the relations than entities in WebNLG, as opposed to what we observed in NYT. We attribute such difference to the different number of relations contained in the two datasets (i.e., 24 in NYT and 246 in WebNLG), which makes the identification of relation much harder in WebNLG.
C Supplemental Experiments
In addition to validating the effectiveness of the proposed CASREL framework in handling the overlapping triple problem, we also conduct a set of supplemental experiments to show the generalization capability in more general cases on four widely used datasets, namely, ACE04, NYT10-HRL, NYT11-HRL and Wiki-KBP. Unlike the datasets we adopt in the main experiment, most test sentences in these datasets belong to the Normal class where no triples overlap with each other. Table 5 shows the result of a comprehensive comparison with recent state-of-the-art methods.
Notably, there are two different evaluation metrics selectively adopted among previous works: (1) The widely used one is Partial Match as we described in Section 4.1, i.e., an extracted relational triple (subject, relation, object) is regarded as correct only if the relation and the heads of both subject and object are all correct (Li and Ji, 2014; Miwa and Bansal, 2016; Katiyar and Cardie, 2017; Zheng et al., 2017; Zeng et al., 2018; Takanobu
<sup>5</sup>Available at: https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip