SlowGuess/NAACL_2021_OCR huggingface dataset:
Fenglong Ma, Yaliang Li, Chenwei Zhang, Jing Gao, Nan Du, and Wei Fan. 2019. Mcvae: Margin-based conditional variational autoencoder for relation classification and pattern generation. In The World Wide Web Conference, WWW '19, page 3041-3048, New York, NY, USA. Association for Computing Machinery.
Diego Marcheggiani and Ivan Titov. 2016. Discrete-state variational autoencoders for joint discovery and factorization of relations. Transactions of the Association for Computational Linguistics, 4:231-244.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Tapas Nayak and Hwee Tou Ng. 2020. Effective modeling of encoder-decoder architecture for joint entity and relation extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8528-8535.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018a. DSGAN: Generative adversarial training for distant supervision relation extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505, Melbourne, Australia. Association for Computational Linguistics.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018b. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2137-2147, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Machine Learning and Knowledge Discovery in Databases*, pages 148–163, Berlin, Heidelberg. Springer Berlin Heidelberg.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning Internal Representations by Error Propagation, page 318-362. MIT Press, Cambridge, MA, USA.
Heng She, Bin Wu, Bai Wang, and Renjun Chi. 2018. Distant supervision for relation extraction with hierarchical attention and entity descriptions. In International Joint Conference on Neural Networks (IJCNN), pages 1-8.
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 455-465, Jeju Island, Korea. Association for Computational Linguistics.
Jakub Tomczak and Max Welling. 2018. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223.
Bayu Distiawan Trisedya, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. GTR-LSTM: A triple encoder for sentence generation from RDF data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1627-1637, Melbourne, Australia. Association for Computational Linguistics.
Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural relation extraction for knowledge base enrichment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 229-240, Florence, Italy. Association for Computational Linguistics.
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018. RESIDE: Improving distantly-supervised neural relation extraction using side information. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1257-1266, Brussels, Belgium. Association for Computational Linguistics.
Guanying Wang, Wen Zhang, Ruoxu Wang, Yalin Zhou, Xi Chen, Wei Zhang, Hai Zhu, and Huajun Chen.
2018. Label-free distant supervision for relation extraction via knowledge graph embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2246-2255, Brussels, Belgium. Association for Computational Linguistics.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019a. Topic-guided variational autoencoders for text generation. arXiv preprint arXiv:1903.07137.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019b. Kepler: A unified model for knowledge embedding and pretrained language representation. arXiv preprint arXiv:1911.06136.
Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1366-1371, Seattle, Washington, USA. Association for Computational Linguistics.
Shanchan Wu, Kai Fan, and Qiong Zhang. 2019. Improving distantly supervised relation extraction with neural noise converter and conditional optimal selector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7273-7280.
Yi Wu, David Bamman, and Stuart Russell. 2017. Adversarial training for relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1778-1783, Copenhagen, Denmark. Association for Computational Linguistics.
Peng Xu and Denilson Barbosa. 2019. Connecting language and knowledge with heterogeneous representations for neural relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3201-3206, Minneapolis, Minnesota. Association for Computational Linguistics.
Kaijia Yang, Liang He, Xin-yu Dai, Shujian Huang, and Jiajun Chen. 2019. Exploiting noisy data in distant supervision relation classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3216-3225, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Distant supervision relation extraction with intra-bag and inter-bag attentions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2810-2819, Minneapolis, Minnesota. Association for Computational Linguistics.
Yujin Yuan, Liyuan Liu, Siliang Tang, Zhongfei Zhang, Yueting Zhuang, Shiliang Pu, Fei Wu, and Xiang Ren. 2019. Cross-relation cross-bag attention for distantly-supervised relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 419-426.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1753-1762, Lisbon, Portugal. Association for Computational Linguistics.
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3016-3025, Minneapolis, Minnesota. Association for Computational Linguistics.
Yijia Zhang and Zhiyong Lu. 2019. Exploring semi-supervised variational autoencoders for biomedical relation extraction. Methods, 166:112 - 119. Deep Learning in Bioinformatics.
Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis. 2020. Dgl-ke: Training knowledge graph embeddings at scale. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 739-748, New York, NY, USA. Association for Computing Machinery.
Yaoming Zhu, Juncheng Wan, Zhiming Zhou, Liheng Chen, Lin Qiu, Weinan Zhang, Xin Jiang, and Yong Yu. 2019. Triple-to-text: Converting rdf triples into high-quality natural languages via optimizing an inverse kl divergence. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 455-464.
A.1 The NYT10 Dataset
As described in Bai and Ritter (2019), the NYT10 dataset has been released in several versions. The original one, follows the setting of Riedel et al. (2010), where two sets of data were created. Later versions (Lin et al., 2016) merged the two sets in order to construct a larger dataset. This merging resulted into 570,300 instances for training. However, in this version of the data exist overlaps in pairs between the training and the test set. The amount of overlaps is significant and accounts for
47, 477 instances, which is approximately  $27.5\%$  of the testing instances. The version was corrected later on but there still remain methods that use the non-filtered data. Recently, Han et al. (2019) released a finalised version removing the overlaps, resulting in 522, 611 total training instances. In our experiments we evaluate the proposed model on both versions.
It is also important to note that NYT10 has been used by the community in two settings: bag-level and sentence-level. In the bag-level setting, a pair's relation is defined based on a bag of sentences that contain the pair. On the contrary, in the sentence-level setting a pair's relation is predicted for each sentence. Training data are obtained using distant supervision, while test data are manually annotated (Hoffmann et al., 2011).
A.2 Data Pre-processing Details
We found that the dataset includes several duplicate instances, i.e. the exact same sentence with the exact same pair. We remove such cases from our training data since they can bias the training process. However, they are preserved on the validation and test sets for a fair comparison with other methods. We convert the dataset to lowercase and replace all digits with the hash character (#). We randomly select  $10\%$  of the training bags as our validation set.
Train	Validation	Test
Processed	Instances	400,100	53,319	172,448
Bags	248,352	28,108	96,678
Facts	16,338	1,823	1,950
Negatives	233,092	26,301	94,917
Instances	469,290	53,321	-
Bags	252,044	28,109	-
Duplicates	62,327	-	-
Outliers	5,570	-	-
Table 6: Statistics of the NYT10 (520K version) dataset.
Sentence Length Filtering. We restrict the length of a sentence to 50 words for the NYT10 dataset and to 30 for the WIKIDISTANT dataset. If at least one of the arguments of a pair is located in a span after the maximum sentence length, then the sentence is resized to contain the words from the first argument until the second. We also add a maximum number of 5 words to the left and 5 words to the right if the total length allows. If the length of the resized sentence is still larger than
Train	Validation	Test
Processed	Instances	434,453	62,333	172,448
Bags	258,843	29,303	96,678
Facts	17,387	1,942	1,950
Negatives	242,644	27,374	94,917
Instances	507,755	-	-
Bags	262,649	-	-
Duplicates	66,130	-	-
Outliers	5,856	-	-
Table 7: Statistics of the NYT10 (570K version) dataset.
Train	Validation	Test
Processed	Instances	1,000,765	29,145	28,897
Bags	572,215	14,748	15,509
Facts	201,356	4,333	4,333
Negatives	370,859	10,415	11,176
Instances	1,050,246	-	-
Bags	575,620	-	-
Duplicates	43,978	-	-
Outliers	5,503	-	-
the maximum sentence length, the sentence is removed from the training set. The reason for this choice is that we want to construct contextualised argument representations. Without the arguments inside the sentence, such representations cannot be formed. We call such removed sentences outliers. Outliers are not removed for the validation and test sets. Relevant statistics are shown in Tables 6, 7 and 8.
Vocabulary construction. In order to construct the word vocabulary, we use the unique sentences contained in the training set, as resulted from the removal of duplicate instances and the sentence length filtering. Since each sentence in the dataset can contain multiple pairs, it is repeated for each pair. Using non-unique sentences can lead to counting larger frequencies for certain words and producing a misleading vocabulary. We restrict the vocabulary to contain the 40K most frequent words for NYT10, with a coverage of  $97.78\%$  in the training set and to 50K for WIKIDISTANT with a coverage of  $96\%$ . Other words are replaced with the UNK token.
DSRE Models. Table 9 shows the parameters used for training the model on the NYT10 and
WIKIDISTANT dataset. In the VAE setting Adaptive Softmax (Grave et al., 2017) was incorporated instead of regular Softmax for faster training. We used three clusters by splitting the vocabulary in  $\left\lfloor \frac{|V|}{15} \right\rfloor$  and  $\left\lfloor \frac{3|V|}{15} \right\rfloor$  words.
Table 8: Statistics of the WIKIDISTANT dataset.
Parameter	NYT	WIKI
Batch size	128	128
Max bag size	500	500
Learning rate	0.001	0.001
Weight decay	10-6	10-6
Gradient clipping	10	5
Optimiser	Adam	Adam
Early stopping patience	5	5
Task loss weight λ	0.8, 0.9	0.9
Word embedding E(w) dim.	50	50
Relation embedding E(r) dim.	64	128
Position embedding E(p) dim.	8	8
Latent code z dim.	64	64
Teacher force	0.3	0.3
Encoder dim.	256	256
Encoder layers	1	1
Decoder dim.	256	256
Decoder layers	1	1
Input dropout	0.3	0.3
Word dropout	0.3	0.1
Knowledge Base Embeddings. In order to train KB entity embeddings we used the DGL-KE toolkit (Zheng et al., 2020). We use the same set of hyper-parameters for both Freebase and Wikidata as shown in Table 10. For Freebase we select 5,000 triples as the validation set, while for Wikidata we use the validation set provided in the transductive setting (5, 136 triples).
Table 9: Models hyper-parameters for each dataset.
Parameter	Value
Model	TransE_12
Emb. size	64
Max train step	500,000
Batch size	1024
Negative sample size	256
Learning rate	0.1
Gamma	10.0
Negative adversarial sampling	True
Adversarial temperature	1.0
Regularisation coefficient	10-7
Regularisation norm	3
Table 10: Knowledge Base Embeddings hyperparameters.
A.4 WIKIDISTANT Relation Categories
Since WIKIDISTANT contains 454 relations, their labels are used directly from the WikiData properties<sup>9</sup>. Here, we add explanations about the top 10 most frequent categories used in Figures 4c, 4d.
P17	country
P3373	sibling
P131	located in the administrative territorial entity
P54	member sports team
P175	performer
P161	cast member
P361	part of
P50	author
P150	contains administrative territorial entity
P527	has part
Table 11: Explanations of the top 10 most frequent WIKIDISTANT relation categories.
A.5 Additional Plots
Figure 5 illustrates the t-SNE plot of the latent space for the NYT10 validation set. We observe similar clusters to that of the KB (Figure 4a).
Figure 6 illustrates the PR-curves for the non-filtered version of the NYT10 dataset (570K). Here, KB-priors perform comparably with Normal prior but mostly improve the tail of the distribution (after  $50\%$  of the recall range). We could not obtain the PR curve for the JOINTNRE method, thus it is not present in the figure.
$^9$ https://www.wikidata.org/wiki/Wikidata: List_of_property
Figure 5: t-SNE plot of the latent vector  $(\mu)$  for the NYT10 (520K) validation set, when using KB priors during training.
Figure 6: Precision-Recall curves for the NYT10 (570K) test set.