Venue: NAACL 2022

# Does it Really Generalize Well on Unseen Data?

Systematic Evaluation of Relational Triple Extraction Methods

Juhyuk Lee1, Min-Joong Lee2, June Yong Yang3, Eunho Yang3

Samsung Research, Samsung Electronics, South Korea1,

Samsung Advanced Institute of Technology, Samsung Electronics, South Korea2,

Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

{juhyuk.lee, minjoong.lee}@samsung.com

{laoconeth, eunhoy}@kaist.ac.kr

This work was done when Juhyuk Lee was with KAIST as a student.Equal contribution.

Footnote 1: footnotemark:

###### Abstract

The ability to extract entities and their relations from unstructured text is essential for the automated maintenance of large-scale knowledge graphs. To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before. In this paper, we show that although existing extraction models are able to easily memorize and recall already seen triples, they cannot generalize effectively for unseen triples. This alarming observation was previously unknown due to the composition of the test sets of the go-to benchmark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance. To separately measure the generalization performance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets. In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models.

+
Footnote †: Equal contribution.

+
Footnote †: Equal contribution.

## 1 Introduction

Relational Triple Extraction (RTE), a more generalized version of Relation Extraction, is the task of extracting all relational triples in the form of _(subject, relation, object)_ from a given sentence. The ability to extract such triples is much required in the construction and maintenance of knowledge graphs such as DBpedia Auer et al. (2007), Freebase Bollacker et al. (2008), and Wikidata Vrandecic and Krotzsch (2014) from documents containing a large number of new and emerging information.

With language model pretraining Devlin et al. (2019); Radford et al. (2019), RTE methods achieved a new state-of-the-art Wei et al. (2020); Wang et al. (2020); Zheng et al. (2021). However, whether the performance of these methods attributes to their capabilities of recalling already seen data or their ability to generalize and extract relations from unseen data is yet to be scrutinized.

To separately evaluate memorization and generalization, we categorize the triples in the test set into three types: _entirely seen_ (completely overlaps with triples in their respective training sets), _partially seen_ (overlaps partially), and _unseen_ (completely new). We analyze common RTE benchmark datasets NYT Riedel et al. (2010) and WebNLG Gardent et al. (2017) using these categories, and find that 89.61% and 91.10% of triples in NYT and WebNLG _test sets_ are of the _entirely seen_ type. This suggests that benchmark results on these datasets are heavily biased towards recalling seen data. Thus, more reliable systematic evaluation methods are in need to test generalization performance.

In this paper, we propose three natural strategies for evaluating generalization performance from a limited number of given _partially seen_ and _unseen_ triples. For the first two strategies, we directly increase the proportion of _partially seen_ and _unseen_ triples in test sets by 1) rearranging their respective datasets or 2) sifting out instances in their respective training sets that overlap with the test set, rendering them unobserved. For the last strategy, we 3) augment test sets by replacing entities in each test instance with similar (and probably not pre-observed) words in order to increase diversity as well as the proportion of _partially seen_ and _useen_ triples. In addition to evaluating recent RTE methods with the above evaluation strategies, we propose a simple yet effective augmentation technique called _Entity Noising_ to help RTE methods to generalize beyond training data.



Our contributions are:

* We show for the first time that the current benchmark datasets for relational triple extraction exhibit significant entity pair overlap between training and test data.
* We confirm that the current state-of-the-art models trained on such datasets cannot generalize well to unseen triples.
* We propose three evaluation strategies to evaluate RTE methods systematically, and show that the proposed simple augmentation technique called _Entity Noising_ can assist RTE methods in generalizing to unseen data.

## 2 Fine-grained Re-evaluation of the Current State-of-the-arts

In this section, we mainly scrutinize the generalization capabilities of current Relational Triple Extraction (RTE) methods and show for the first time that they indeed struggle in extracting relational triples from the context for unseen cases.

### Datasets and Evaluation Metrics

We use two well-known benchmark datasets NYT Riedel et al. (2010) and WebNLG Gardent et al. (2017) for evaluation, following Wang et al. (2020) and Zheng et al. (2021). Also, predicted triples are considered correct only if their whole entity spans of both subject and object and their relation are exactly matched with ground truth. We report the standard micro F1 for the overall performance.

To assess the memorization and generalization performances separately, we also compute type F1 with three triple types: _entively seen_, _partially seen_, and _unseen_ (Section 2.2). Type F1 is nothing but F1 evaluated using instances which only consist of a single triple type.

### Triple Types

We describe three triple types - _entirely seen_, _partially seen_, and _unseen_ - in detail. For a set of triples in the training set \(S=\{(s_{i},r_{i},o_{i})\}^{n}_{i=1}\), the type of each triple \((s,r,o)\) in the test set are defined as follows. A triple \((s,r,o)\) belongs to the _entirely seen_ type if \((s,r,o)\in S\). For _parially seen_ type, triples \((s,r,o)\) which satisfy conditions \([(s,r,\cdot)\in S\) or \((\cdot,r,o)\in S]\) and \((s,r,o)\not\in S\) belong to it. Other triples belong to _unseen_ type.

### Detailed Evaluation with Triple Types

Using type F1, we show that the current state-of-the-arts CasRel Wei et al. (2020), TPLinker Wang et al. (2020), and PRGC Zheng et al. (2021) are only able to memorize and recall already seen triples, and are unable to generalize effectively for unseen triples (See Table 2). This observation was previously unknown due to the overlaps between training and test data of benchmark datasets NYT and WebNLG.

Indeed, as shown in Table 1, 89.61% and 91.10% of triples in NYT and WebNLG _test sets_ completely overlap with triples in their respective training sets (such triples are defined as _entirely seen_ type), while _partially seen_ and _unseen_ samples that require generalization to predict are but a small portion.

## 3 Evaluating Generalization Performance

As shown in Table 1, the proportion of _partially seen_ and _unseen_ triples in the original benchmark test sets are so small that they are not diverse

\begin{table}
\begin{tabular}{l c r r r r r r r r r r} \hline \hline \multirow{2}{*}{**Triple type**} & \multicolumn{5}{c}{NYT} & \multicolumn{5}{c}{WebNLG} \\ \cline{2-14}  & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 & Aug. & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 & Aug. \\ \hline Entirely seen (\%) & **89.61** & 14.20 & 63.24 & 55.45 & 49.27 & 5.76 & **91.10** & 45.47 & 78.03 & 56.50 & 39.20 & 17.21 \\ Partially seen (\%) & 8.64 & 66.72 & 31.56 & 38.09 & 43.19 & 46.33 & 7.47 & 34.20 & 17.05 & 30.86 & 37.40 & 36.17 \\ Unseen (\%) & 1.75 & 19.08 & 5.20 & 6.46 & 7.54 & 47.91 & 1.43 & 20.33 & 4.92 & 12.63 & 23.40 & 46.62 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Triple type statistics of original test sets, _rearranged_, _overlap sifted_ datasets, and _augmented test sets_.

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline  & **Method** & F1 & Entire & Partial & Unseen \\ \cline{2-6}  & CasRel & 90.1 (89.0*) & 93.8 & 64.6 & 45.4 \\  & TPLinker & 92.4 (92.0†) & 96.0 & 65.9 & 50.3 \\  & PRGC & 89.1 (92.7†) & 92.9 & 65.4 & 44.5 \\ \hline \multirow{2}{*}{**Pixel**} & CasRel & 88.3 (86.4‡) & 92.0 & 54.3 & 45.5 \\  & TPLinker & 89.0 (86.7†) & 92.6 & 62.6 & 56.0 \\  & PRGC & 88.0 (88.5†) & 92.1 & 56.2 & 34.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: F1 and type F1 of recent RTE methods. Results with † marks are from their papers. Results with ‡ marks are reported by Ren et al. (2021). Other results are our reproductions using official implementations.

 

enough, rendering the evaluations of generalization capabilities unreliable. Equipped with this observation, we propose three strategies to increase the proportion of _partially seen_ and _unseen_ triples and add diversity to them for reliable evaluation of Relational Triple Extraction (RTE) methods.1

Footnote 1: Three versions of datasets can be found in [https://github.com/sehkmg/rte-eval](https://github.com/sehkmg/rte-eval).

### Rearranged Dataset

The basic approach to increasing the proportion of _partially seen_ and _unseen_ triples is to rearrange the given dataset splits. However, it is not possible to emphasize unseen data just by randomly rearranging the dataset, since it inadvertently incurs overlaps between training and test data2.

Footnote 2: We are only able to emphasize unseen data to at most \(2\%\) with \(10^{6}\) random trials.

To emphasize the unseen data, we repeatedly select a triple and distribute every instance which contains that triple to the test set, rendering them unobserved in the training set. In order to minimize redundancy in the test set, we select a triple one by one which occurs less. The detailed statistics are shown in Table 1 and Appendix B.

### Overlap Sifted Dataset

We propose another simple strategy to emphasize unseen test samples. To render a triple in the test set unobserved, we remove the instances containing that triple from the training set. Specifically, we randomly choose \(k\)% of the unique triples from the test set, then remove all the instances containing the selected triples from the _training set_ to construct an _overlap sifted dataset_. For demonstration, we construct three such datasets by choosing \(k=5,10,15\)%, respectively. The detailed statistics are presented in Table 1 and Appendix B.

### Augmented Test Set

To add more diversity to _partially seen_ and _unseen_ samples as well as increasing their proportion, we create an _augmented test set_. The key idea is to substitute every entity defined in every triple with probable alternative words by utilizing the knowledge of Masked Language Models Radford et al. (2019); Devlin et al. (2019) and GloVe word embeddings Pennington et al. (2014), similar to the data augmentation technique used in Jiao et al. (2020). With the _augmented test set_, it is able to assess whether the ability of an RTE method is influenced by the authenticity of the given text3. The details are in Appendix C and statistics are present in Table 1 and Appendix B.

Footnote 3: An ideal RTE model should be able to extract the relational triple (The [United States] President [Christopher]) if such fictitious content happens to exist in the given text.

## 4 Entity noising

We further propose _Entity Noising_, a simple augmentation technique to enhance the generalization performance of existing Relational Triple Extraction methods. The key idea of _Entity Noising_ is to replace the entities in the given training input sentence with completely random noisy words. To apply Entity Noising, we sample a random noisy word \(w^{\prime}\) for each entity \(w\), i.e., \(w^{\prime}\sim P(w^{\prime}\mid w)\). The sampling strategy is defined as follows. First,

\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline \hline  & **Method** & \multicolumn{8}{c}{Original} & \multicolumn{8}{c}{Rearranged} \\ \cline{3-17}  & Prec. & Rec. & F1 & Entire & Partial & Unseen & Prec. & Rec. & F1 & Entire & Partial & Unseen \\ \cline{2-17}  & CasRel & 90.2 & **90.0** & **90.1** & **93.8** & 64.6 & **45.4** & **65.9** & **60.1** & **62.9** & **85.8** & **65.0** & 42.3 \\  & CasRel+EN & **91.6** & 88.8 & **90.1** & 93.7 & **65.0** & 44.8 & 65.2 & 59.3 & 62.1 & 81.1 & 64.9 & **44.0** \\ \multirow{4}{*}{**CIF**} & TPLinker & **92.3** & **92.5** & **92.4** & **96.0** & 65.9 & 50.3 & 69.0 & **60.8** & **64.7** & 83.3 & **66.7** & 46.8 \\  & TPLinker+EN & 92.2 & 91.8 & 92.0 & 95.5 & **66.0** & **54.4** & **69.2** & 60.3 & 64.5 & **84.2** & 66.3 & **47.2** \\ \cline{2-17}  & PRGC & 88.4 & **89.9** & **89.1** & **92.9** & **65.4** & 44.5 & 63.5 & **61.6** & **62.6** & **81.6** & **64.2** & 45.1 \\  & PRGC+EN & **89.1** & 88.7 & 88.9 & 92.3 & **65.4** & **51.2** & **63.9** & 60.6 & 62.2 & 79.8 & **64.2** & **46.2** \\ \cline{2-17}  & CasRel & **90.1** & 86.6 & **88.3** & **92.0** & **54.3** & 45.5 & **73.6** & **64.2** & **68.6** & **89.6** & 52.3 & 41.5 \\  & CasRel+EN & 88.8 & **86.8** & 87.8 & 91.3 & 48.9 & **53.8** & 72.5 & 63.2 & 67.5 & 85.7 & **54.0** & **45.8** \\ \multirow{4}{*}{**DIF**} & TPLinker & **90.2** & **87.7** & **89.0** & **92.6** & **62.6** & 56.0 & **75.1** & 63.9 & 69.1 & 88.5 & 52.7 & 42.9 \\  & TPLinker+EN & 89.3 & 87.4 & 88.3 & 91.8 & 60.0 & **71.4** & 73.5 & **66.2** & **69.7** & **88.7** & **53.6** & **49.3** \\ \cline{2-17}  & PRGC & **89.7** & **86.4** & **88.0** & **92.1** & 56.2 & 34.5 & 61.6 & 62.0 & 61.8 & 79.2 & 47.2 & 28.3 \\ \cline{2-17}  & PRGC+EN & 87.6 & 85.4 & 86.5 & 90.2 & **57.5** & **40.0** & **68.0** & **62.5** & **65.2** & **82.8** & **52.8** & **34.4** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results of recent RTE methods with and without _Entity Noising_ on original and _rearranged_ datasets. Every result are our reproduction.

 

we sample token length \(l^{\prime}\in\left\{l-1,l,l+1\right\}\) of \(w^{\prime}\) with probability \(P(l^{\prime}=l)=p_{en}^{len}\) and \(P(l^{\prime}=l-1)=P(l^{\prime}=l+1)=\left(1-p_{en}^{len}\right)/2\), where \(l\) is a token length of \(w\). This sampling process introduces a small(\(\pm 1\)) perturbation to the token length \(l\) to prevent the model from memorizing the number of tokens. After sampling \(l^{\prime}\), we sample \(w^{\prime}\) from the uniform distribution \(w^{\prime}\sim\mathrm{Uniform}(V_{l^{\prime}})\), where \(V_{l^{\prime}}\) is a subset of the vocabulary \(V\) which consists of all words of token length \(l^{\prime}\).

With sampling strategy \(w^{\prime}\sim P(w^{\prime}\mid w)\), Entity Noising is applied to a given training sentence \(\mathbf{x}_{\text{original}}=(w_{1},w_{2},\cdots,w_{K})\) to produce a noised sentence \(\mathbf{x}_{\text{noised}}=(w_{1}^{\prime},w_{2}^{\prime},\cdots,w_{K}^{\prime})\) according to the following rule:

\[w_{k}^{\prime}=\begin{cases}w_{k}^{\prime}\sim P(w_{k}^{\prime}\mid w_{k}),& \text{if $w_{k}$ is an entity}\\ w_{k},&\text{otherwise}\end{cases}\]

Finally, we determine which input \(\mathbf{x}\) is fed to the extractor model with probability \(P(\mathbf{x}=\mathbf{x}_{\text{noised}})=p_{en}\) and \(P(\mathbf{x}=\mathbf{x}_{\text{original}})=1-p_{en}\). An overview illustration of _Entity Noising_ is shown in Figure 1.

_Entity Noising_ is different from a commonly used data augmentation method such as Wei and Zou (2019) which replaces entities with words similar to them. _Entity Noising_ replaces entities with completely random noisy words. This feature allows the model to utilize entity-agnostic information, so that the model can learn to extract triples from sentences by focusing on the context information rather than the entities themselves. Therefore, with _Entity Noising_, the model is kept away from memorizing the entity pair along with its relation.

## 5 Experiments

We conduct a series of experiments with recent Relational Triple Extraction (RTE) methods on newly constructed datasets (Section 3).

Figure 1: Overview of _Entity Noising_.

 

Rearranged Dataset (Section 3.1)Table 3 shows the lack of generalization capabilities of recent RTE methods in _rearranged datasets_ as well as original datasets. On _rearranged datasets_, _Entity Noising_ consistently improves the ability of generalization on _unseen_ triples, and for _partially seen_ triples, it at least does not hurt the generalization capabilities. For original datasets, the evaluation can be biased on some specific _partially seen_ and _unseen_ samples since their proportion in test sets is small, rendering inconsistent results.

Overlap Sifted Dataset (Section 3.2)With _overlap sifted datasets_ and original datasets, we evaluate recent RTE methods with and without _Entity Noising_ to get more insight into what extent they generalize on unseen data. Table 4 shows that recent RTE methods struggle in extracting triples from unseen data, while _Entity Noising_ promotes their generalization capabilities in most cases.

Augmented Test Set (Section 3.3)To assess whether the ability of an RTE method is influenced by the authenticity of the given text, we evaluate recent RTE methods with and without _Entity Noising_ on _augmented test set_. We find that current RTE methods are substantially influenced by the authenticity of the given text, while _Entity Noising_ relieves that influence by a huge margin (See Table 5).

## 6 Related Work

Open Information Extraction (Open IE)Open IE is the task of extracting relations from the given text without predefined relation type Stanovsky et al. (2018); Zhan and Zhao (2020); Cui et al. (2018); Kolluru et al. (2020). Although Open IE is a more general task than Relational Triple Extraction, it is necessary to extract information using fixed relation type to get high quality relational triples from specific domains such as science and business.

Data Leakage in NLPThe overlapping problem between training and test data makes the evaluation biased towards assessing memorization capabilities of models. Several works point out the overlapping problem and quantify data leakage in basic NLP tasks Elangovan et al. (2021) and Open-Domain Question Answering Lewis et al. (2021), but Relational Triple Extraction was not considered yet.

## 7 Conclusion

In this paper, we disclosed for the first time that recent Relational Triple Extraction (RTE) methods struggle to extract triples from unseen data, which was previously unknown due to the test-train overlap problem in popular benchmark datasets. To properly assess the generalization capabilities of RTE methods, we developed three strategies to construct _rearranged dataset_, _overlap sifted dataset_, and _augmented test set_ from original datasets. Furthermore, we proposed a simple yet effective noising method to promote generalization and experimentally confirm that it effectively improves the generalization capabilities of existing RTE methods.

### Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075 Artificial Intelligence Graduate School Program(KAIST), No.2019-0-01371 Development of Brain-inspired AI with Human-like Intelligence) and National Research Foundation of Korea (NRF) grant (2018R1A5A1059921).

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & **Method** & Prec. & Rec. & F1 \\ \cline{2-5}  & CasRel & 39.6 & 22.4 & 28.6 \\  & CasRel+EN & **54.3** & **34.5** & **42.2** \\ \multirow{4}{*}{**Source**} & TPLinker & 44.5 & 22.6 & 30.0 \\  & TPLinker+EN & **56.2** & **34.7** & **42.9** \\ \cline{2-5}  & PRGC & 37.2 & 25.4 & 30.2 \\  & PRGC+EN & **51.8** & **28.1** & **36.4** \\ \hline \multirow{4}{*}{**Source**} & CasRel & 66.9 & 32.1 & 43.4 \\  & CasRel+EN & **70.4** & **53.6** & **60.9** \\ \cline{1-1} \cline{2-5}  & TPLinker & 69.6 & 39.1 & 50.1 \\ \cline{1-1}  & TPLinker+EN & **73.4** & **55.2** & **63.0** \\ \cline{1-1} \cline{2-5}  & PRGC & 67.5 & 42.0 & 51.8 \\ \cline{1-1}  & PRGC+EN & **69.0** & **56.3** & **62.0** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of recent RTE methods with and without _Entity Noising_ on _augmented test sets_.

 



## References

* Auer et al. (2007) Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. ISWC'07/ASWC'07, Berlin, Heidelberg. Springer-Verlag.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
* Cui et al. (2018) Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 407-413. Melbourne, Australia. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Elangovan et al. (2021) Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1325-1335, Online. Association for Computational Linguistics.
* Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for NLG micro-planners. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 179-188, Vancouver, Canada. Association for Computational Linguistics.
* Jiao et al. (2020) Xiaqoi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4163-4174, Online. Association for Computational Linguistics.
* Kolluru et al. (2020) Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Mausam, and Soumen Chakrabarti. 2020. IIMoJIE: Iterative memory-based joint open information extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5871-5886, Online. Association for Computational Linguistics.
* Lewis et al. (2021) Patrick Lewis, Pontus Steaterorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1000-1008, Online. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Ren et al. (2021) Feiliang Ren, Longhui Zhang, Shujuan Yin, Xiaofeng Zhao, Shilei Liu, and Bochao Li. 2021. _A Conditional Cascade Model for Relational Triple Extraction_, page 3393-3397. Association for Computing Machinery, New York, NY, USA.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_, pages 148-163. Springer.
* Stanovsky et al. (2018) Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 885-895, New Orleans, Louisiana. Association for Computational Linguistics.
* Vrandecic and Krotzsch (2014) Denny Vrandecic and Markus Krotzsch. 2014. Wikidata: A free collaborative knowledgebase. _Commun. ACM_, 57(10):78-85.
* Wang et al. (2020) Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1572-1582.
* Wei and Zou (2019) Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
* Wei et al. (2020) Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1476-1488, Online. Association for Computational Linguistics

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remin Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Ph, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhan and Zhao (2020) Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction on accurate corpus. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 9523-9530. AAAI Press.
* Zheng et al. (2021) Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Xu Ming, and Yefeng Zheng. 2021. PRGC: Potential relation and global correspondence based joint relational triple extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6225-6235. Association for Computational Linguistics.

 

Training Details

In general, we train CasRel, TPLinker, and PRGC for 300, 500 epochs on NYT, WebNLG datasets. It takes 5 GPU days for training models on NYT and I GPU day for training models on WebNLG. We select the best model by only using the F1 score of the given validation set except _overlap sifted dataset_. For _overlap sifted dataset_, the training instances are sifted out according to the test instances, rendering the triple type statistics of valid and test sets are different. Therefore, we select the best model by using the F1 score of _overlap sifted_ test sets. For Entity Noising, we set \(p_{en}\) to 0.1 and 0.05 for NYT and WebNLG datasets and set \(p^{len}_{en}\) to 0.4. Every model is based on pre-trained BERT model _BERTbase-cased_ from Huggingface Transformers Wolf et al. (2020), which contains 110M parameters.

## Appendix B Dataset Statistics

The statistics of dataset split are shown in Table 6. To compute type F1 defined in Section 2.1, stratification is necessary by extracting test instances which only consist of single triple type among _entirely seen_, _partially seen_ and _unseen_. The stratification statistics are shown in Table 7.

## Appendix C Augmented test sets

Discussions on augmented test setIt is worthy to note that the samples in the _augmented test set_ may not be "true" statements in the real world but rather invented, as by construction their entities are replaced with other similar words (See examples in Figure 2). However, the true meaning of the entity words is fundamentally irrelevant to the relation between them given the context. Also, it is unknown whether the relation in the sentence is a fact. Thus, the ability of an RTE model to extract relational triples should not be influenced by the authenticity of the given text. Note that an ideal RTE model should be able to extract the relational triple (The [United States] President [Christopher]) if such fictitious content happens to exist in the given text.

Although the ideal RTE model should not be influenced by the authenticity of the given text, there exists potential risk. It is that the deployed RTE model might extract the invalid triple from wrong text. Therefore, the validation process which checks the triple is needed before adding it to the knowledge graph.

Construction details of augmented test setWe now describe the construction details of the augmented test set. First, we preemptively run the language tokenizer to flag the wordpieces in the entity words. we substitute all entity words in the triples with masks (one mask _per word_, not _per wordpiece_). For single-word-single-wordpiece entities, we use the language model to fill in their masks independently. For single-word-multi-piece entities, we do not use the language model but search and substitute for the k-nearest words of the original entity word in the GloVe embedding space. For multi-word entities, each word constituting an entity is sequentially substituted using the language model.

Now we describe the detailed construction of \(T_{\text{Augmented}}\). To measure the generalization performance properly, it is required that the _augmented test set_\(T_{\text{Augmented}}\) consists of _partially seen_ triples as well as _unseen_ triples since the ideal RTE model is required to effectively extract both _partially seen_ and _unseen_ triples. Therefore, we first construct four augmented components of the test set \(T_{\text{ss}}\), \(T_{\text{su}}\), \(T_{\text{us}}\), \(T_{\text{uu}}\) and take a union of them to create the final _augmented test set_\(T_{\text{Augmented}}=T_{\text{ss}}~{}\cup~{}T_{\text{su}}~{}\cup~{}T_{ \text{uus}}~{}\cup~{}T_{\text{uu}}\). Among the four components, \(T_{\text{ss}}\) consists of triples with seen subject and object; \(T_{\text{su}}\) consists of triples with seen subject and unseen object; \(T_{\text{us}}\) is symmetrical with \(T_{\text{su}}\); \(T_{\text{uu}}\) consists of triples with unseen subject and object.

We now describe the construction details of four components: \(T_{\text{ss}}\), \(T_{\text{su}}\), \(T_{\text{uss}}\) and \(T_{\text{uu}}\). First, for each sample in the test set \(t^{i}_{\text{Standard}}\in T_{\text{Standard}}\), we get a set of top-\(k\) similar entities \(E^{ij}_{s}\) for each entity \(e^{ij}\) in \(t^{i}_{\text{Standard}}\) independently, so that there is no correlation between each \(E^{ij}_{s}\). Then, we uniformly sample \(e^{ij}_{s}\) from \(E^{ij}_{s}\) and replace \(e^{ij}\) with \(e^{ij}_{s}\) to get \(t^{i}_{\text{Augmented}}\in T_{\text{Augmented}}\).

Construction of \(\mathbf{T_{ss}}\)\(T_{\text{ss}}\) mainly consists of triples in which both subject and object entities are already seen in the training set. Therefore, every subject and object entity \(e^{ij}_{s}\) is sampled from \(E^{ij}_{s}\cap E_{\text{Train}}\) uniformly, where \(E_{\text{Train}}\) is a set of entities appeared in the training set. If we encounter to sample from an empty set, we assign \(e^{ij}_{s}=e^{ij}\).

Construction of \(\mathbf{T_{su}},\mathbf{T_{us}}\)\(T_{\text{su}}\)mainly consists of triples in which subject entities are seen and object entities are unseen in the training set. Therefore, subject and subject/object entities \(e^{ij}_{s}\) are sampled from \(E^{ij}_{s}\cap E_{\text{Train}}\), and object entities \(e^{ij}_{s}\) are 

sampled from \(E_{s}^{ij}\backslash E_{\text{Train}}\) uniformly. \(T_{\text{us}}\) is constructed symmetrically.

**Construction of \(\mathbf{T_{uu}}\quad T_{\textbf{u}}\)** mainly consists of triples in which both subject and object entities are unseen in the training set. Therefore, every subject and object entity \(e_{s}^{ij}\) is sampled from \(E_{s}^{ij}\backslash E_{\text{Train}}\) uniformly.

## Appendix A

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Split**} & \multicolumn{8}{c}{NYT} & \multicolumn{8}{c}{WebNLG} \\ \cline{2-13}  & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 & Aug. & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 & Aug. \\ \hline Train & 56196 & 56196 & 50599 & 47152 & 44003 & - & 5019 & 5019 & 4776 & 3951 & 3193 & - \\ Valid & 5000 & 5000 & 5000 & 5000 & 5000 & - & 500 & 500 & 703 & 703 & 703 & - \\ Test & 5000 & 5000 & 5000 & 5000 & 5000 & 20000 & 703 & 703 & 703 & 703 & 2812 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Dataset statistics of original, _rearranged_, _overlap sifted_ datasets, and _augmented test sets_.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Type**} & \multicolumn{8}{c}{NYT} & \multicolumn{8}{c}{WebNLG} \\ \cline{2-13}  & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 & Ori. & Rearr. & Sift-1 & Sift-2 & Sift-3 \\ \hline Entirely seen & 4292 & 348 & 2733 & 2349 & 2064 & 580 & 155 & 435 & 249 & 160 \\ Partially seen & 473 & 3307 & 1703 & 2027 & 2265 & 42 & 178 & 82 & 133 & 172 \\ Unseen & 88 & 886 & 238 & 262 & 274 & 17 & 174 & 34 & 63 & 99 \\ Others & 147 & 459 & 326 & 362 & 397 & 64 & 196 & 152 & 258 & 272 \\ \hline Total & 5000 & 5000 & 5000 & 5000 & 5000 & 703 & 703 & 703 & 703 & 703 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Stratified test set statistics of original, _rearranged_, and _overlap sifted_ datasets. Each number indicates the number of instances which only consist of respective triple type. Note that an instance can have multiple triples associated with multiple triple types, which are defined with _Others_ type.

 


* [15] M. C. Bresson, A. K. Sridharan, and R. J. Ward. _A survey of the social sciences: a review on the social sciences_. In _2014 IEEE International Conference on Knowledge Discovery and Data Mining_, pages 37-68. ACM, 2014.
* [16] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_, 2014.
* [17] T. L. Chuang, Y. Bengio, and G. Hinton. Deep residual learning with applications to image recognition. _Journal of Machine Learning Research_, 20(1):105009, 2014.
* [18] H. Li, X. Zhang, Z. Liu, and Q. Sun. Deep residual learning with deep convolutional neural networks. In _Advances in Neural Information Processing Systems_, pages 103-113. Springer, 2014.
* [19] W. Lin, N. Yang, and E. Shelhamer. Convolutional neural network for large scale visual recognition. In _NIPS conference on computer vision and pattern recognition_, volume 1, pages 1-10. Springer, 2014.
* [20] I. Goodfellow, I. Sutskever, and I. Goodfellow. Imagenet classification by jointly connected layers. In _NeurIPS conference on computer vision_, pages 138-140. MIT Press, 2014.

[MISSING_PAGE_POST]