SlowGuess/NAACL_2022_OCR huggingface dataset:
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000-1008, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Feiliang Ren, Longhui Zhang, Shujuan Yin, Xiaofeng Zhao, Shilei Liu, and Bochao Li. 2021. A Conditional Cascade Model for Relational Triple Extraction, page 3393-3397. Association for Computing Machinery, New York, NY, USA.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pages 148-163. Springer.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computational Linguistics.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1572-1582.
Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1476-1488, Online. Association for Computational Linguistics.
Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction on accurate corpus. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9523-9530. AAAI Press.
Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Xu Ming, and Yefeng Zheng. 2021. PRGC: Potential relation and global correspondence based joint relational triple extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6225-6235. Association for Computational Linguistics.
In general, we train CasRel, TPLinker, and PRGC for 300, 500 epochs on NYT, WebNLG datasets. It takes 5 GPU days for training models on NYT and 1 GPU day for training models on WebNLG. We select the best model by only using the F1 score of the given validation set except overlap sifted dataset. For overlap sifted dataset, the training instances are sifted out according to the test instances, rendering the triple type statistics of valid and test sets are different. Therefore, we select the best model by using the F1 score of overlap sifted test sets. For Entity Noising, we set  $p_{en}$  to 0.1 and 0.05 for NYT and WebNLG datasets and set  $p_{en}^{len}$  to 0.4. Every model is based on pre-trained BERT model BERT-base-cased from Huggingface Transformers (Wolf et al., 2020), which contains 110M parameters.
The statistics of dataset split are shown in Table 6. To compute type F1 defined in Section 2.1, stratification is necessary by extracting test instances which only consist of single triple type among entirely seen, partially seen and unseen. The stratification statistics are shown in Table 7.
C Augmented test sets
Discussions on augmented test set It is worthy to note that the samples in the augmented test set may not be "true" statements in the real world but rather invented, as by construction their entities are replaced with other similar words (See examples in Figure 2). However, the true meaning of the entity words is fundamentally irrelevant to the relation between them given the context. Also, it is unknown whether the relation in the sentence is a fact. Thus, the ability of an RTE model to extract relational triples should not be influenced by the authenticity of the given text. Note that an ideal RTE model should be able to extract the relational triple (The [United States] President [Christopher]) if such fictitious content happens to exist in the given text.
Although the ideal RTE model should not be influenced by the authenticity of the given text, there exists potential risk. It is that the deployed RTE model might extract the invalid triple from wrong text. Therefore, the validation process which checks the triple is needed before adding it to the knowledge graph.
Construction details of augmented test set We now describe the construction details of the augmented test set. First, we preemptively run the language tokenizer to flag the wordpieces in the entity words. we substitute all entity words in the triples with masks (one mask per word, not per wordpiece). For single-word-single-wordpiece entities, we use the language model to fill in their masks independently. For single-word-multi-piece entities, we do not use the language model but search and substitute for the k-nearest words of the original entity word in the GloVe embedding space. For multi-word entities, each word constituting an entity is sequentially substituted using the language model.
Now we describe the detailed construction of  $T_{\text{Augmented}}$ . To measure the generalization performance properly, it is required that the augmented test set  $T_{\text{Augmented}}$  consists of partially seen triples as well as unseen triples since the ideal RTE model is required to effectively extract both partially seen and unseen triples. Therefore, we first construct four augmented components of the test set  $T_{\text{ss}}$ ,  $T_{\text{su}}$ ,  $T_{\text{us}}$ ,  $T_{\text{uu}}$  and take a union of them to create the final augmented test set  $T_{\text{Augmented}} = T_{\text{ss}} \cup T_{\text{su}} \cup T_{\text{us}} \cup T_{\text{uu}}$ . Among the four components,  $T_{\text{ss}}$  consists of triples with seen subject and object;  $T_{\text{su}}$  consists of triples with seen subject and unseen object;  $T_{\text{us}}$  is symmetrical with  $T_{\text{su}}$ ;  $T_{\text{uu}}$  consists of triples with unseen subject and object.
We now describe the construction details of four components:  $T_{\mathrm{ss}}$ ,  $T_{\mathrm{su}}$ ,  $T_{\mathrm{us}}$  and  $T_{\mathrm{uu}}$ . First, for each sample in the test set  $t_{\mathrm{Standard}}^i \in T_{\mathrm{Standard}}$ , we get a set of top- $k$  similar entities  $E_s^{ij}$  for each entity  $e^{ij}$  in  $t_{\mathrm{Standard}}^i$  independently, so that there is no correlation between each  $E_s^{ij}$ . Then, we uniformly sample  $e_s^{ij}$  from  $E_s^{ij}$  and replace  $e^{ij}$  with  $e_s^{ij}$  to get  $t_{\mathrm{Augmented}}^i \in T_{\mathrm{Augmented}}$ .
Construction of  $\mathbf{T_{ss}}$ $T_{\mathrm{ss}}$  mainly consists of triples in which both subject and object entities are already seen in the training set. Therefore, every subject and object entity  $e_s^{ij}$  is sampled from  $E_{s}^{ij}\cap E_{\mathrm{train}}$  uniformly, where  $E_{\mathrm{train}}$  is a set of entities appeared in the training set. If we encounter to sample from an empty set, we assign  $e_s^{ij} = e^{ij}$
Construction of  $\mathbf{T}_{\mathrm{su}}$ ,  $\mathbf{T}_{\mathrm{us}}$ $T_{\mathrm{su}}$  mainly consists of triples in which subject entities are seen and object entities are unseen in the training set. Therefore, subject and subject/object entities  $e_s^{ij}$  are sampled from  $E_s^{ij} \cap E_{\mathrm{train}}$ , and object entities  $e_s^{ij}$  are
Split	NYT	WebNLG
Ori.	Rearr.	Sift-1	Sift-2	Sift-3	Aug.	Ori.	Rearr.	Sift-1	Sift-2	Sift-3	Aug.
Train	56196	56196	50599	47152	44003	-	5019	5019	4776	3951	3193	-
Valid	5000	5000	5000	5000	5000	-	500	500	703	703	703	-
Test	5000	5000	5000	5000	5000	20000	703	703	703	703	703	2812
Table 6: Dataset statistics of original, rearranged, overlap sifted datasets, and augmented test sets.
Type	NYT	WebNLG
Ori.	Rearr.	Sift-1	Sift-2	Sift-3	Ori.	Rearr.	Sift-1	Sift-2	Sift-3
Entirely seen	4292	348	2733	2349	2064	580	155	435	249	160
Partially seen	473	3307	1703	2027	2265	42	178	82	133	172
Unseen	88	886	238	262	274	17	174	34	63	99
Others	147	459	326	362	397	64	196	152	258	272
Total	5000	5000	5000	5000	5000	703	703	703	703	703
Table 7: Stratified test set statistics of original, rearranged, and overlap sifted datasets. Each number indicates the number of instances which only consist of respective triple type. Note that an instance can have multiple triples associated with multiple triple types, which are defined with Others type.
sampled from  $E_{s}^{ij} \backslash E_{\mathrm{Train}}$  uniformly.  $T_{\mathrm{us}}$  is constructed symmetrically.
Construction of  $\mathbf{T}_{\mathbf{uu}}$ $T_{\mathrm{uu}}$  mainly consists of triples in which both subject and object entities are unseen in the training set. Therefore, every subject and object entity  $e_s^{ij}$  is sampled from  $E_s^{ij} \backslash E_{\mathrm{train}}$  uniformly.
Original Test Samples	Augmented Test Samples
Above the Veil, from Australia, is the third book in a series after Aenir and Castle.	(Above the Veil, precededBy, Aenir)
(Aenir, precededBy, Castle)	Dark Wars Rising, from Australia, is the third book in a series after Sword and Avalon.	(Dark Wars Rising, precededBy, Sword)
(Sword, precededBy, Avalon)
Populous was the architect of 3Arena in Dublin which was completed in December 2008.	(3Arena, location, Dublin)
(3Arena, architect, Populous)	Monolith was the architect of Trinity in Miami which was completed in December 2008.	(Trinity, location, Miami)
(Trinity, architect, Monolith)
Figure 2: Selected examples from WebNLG augmented test set.