SlowGuess/NAACL_2021_OCR huggingface dataset:
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.
Thomas Lin, Mausam, and Oren Etzioni. 2012. Entity linking at web scale. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 84–88, Montreal, Canada. Association for Computational Linguistics.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133, Berlin, Germany. Association for Computational Linguistics.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. 2019. Openceres: When open information extraction meets the semi-structured web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3047-3056, Minneapolis, Minnesota. Association for Computational Linguistics.
Filipe Mesquita, Matteo Cannaviccio, Jordan Schmidek, Paramita Mirza, and Denilson Barbosa. 2019. Knowledgegenet: A benchmark dataset for knowledge base population. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 749-758, Hong Kong, China. Association for Computational Linguistics.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Machine Learning and Knowledge Discovery in Databases*, Lecture Notes in Computer Science, pages 148-163, Berlin, Heidelberg. Springer.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, Atlanta, Georgia. Association for Computational Linguistics.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and I. Dagan. 2018. Supervised open information extraction. In *NAACL-HLT*.
Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2015. Multilingual relation extraction using compositional universal schema.
Patrick Verga, Arvind Neelakantan, and Andrew McCallum. 2017. Generalizing to unseen entities and entity pairs with row-less universal schema. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 613-622, Valencia, Spain. Association for Computational Linguistics.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019. Kepler: A unified model for knowledge embedding and pre-trained language representation. arXiv:1911.06136 [cs].
Dongxu Zhang, Subhabrata Mukherjee, Colin Lockard, Luna Dong, and Andrew McCallum. 2019. Openki: Integrating open information extraction and knowledge bases with relation inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 762-772, Minneapolis, Minnesota. Association for Computational Linguistics.
A Observations and Discussion
It is worth noting that the learned mixing weights between scoring components have the effect of learning the margin for the max-margin loss. We observed that these weights all increased during inference to values from 7 to 10, resulting in an effective margin of 0.10 to 0.14. Setting and fixing the margin (so the model learns relative weighting, but cannot scale the whole model) however did not perform as well, indicating that this learned scaling helped the model to navigate the parameter space.
The bert dual models with average entity text encoding with descriptions on reverb data performed better (by  $0.5\%$ ) on the development split, but worse on the test split, indicating an element of overfitting. With CLS token encoding, dev (and test) results with and without descriptions were not notably different ( $< 0.1\%$ ). FastText development set results were around  $2.5\%$  better with descriptions, which is similar to test results.
B Choice of Model Parameters:
Due to large run times, substantial parameter tuning was not practical. We chose 100 dimensional embeddings as a reasonable compromise between improved expressiveness and increased run times. We found that in most cases the parameters used in (Zhang et al., 2019) (batch size 128, lr 0.005 etc...— see Section 4) were able to produce reasonable results from model checkpoints with the best development set performance. We typically ran two models and chose the best dev set performance from the two. We found that larger batch sizes were more stable in training (ie: less variation in loss and dev set performance between epochs), however lacked the inherent exploration of the parameter space provided by those variations and hence resulted in lower (best epoch) development set performance.
C Details of NYT Data Preparation
The original NYT data has of two splits: training data from 2005-2006 and evaluation data from 2007. We further randomly split the training data into train and development subsets such that entity pairs in the development subset are not present in the training subset and further exclude test set entity pairs from training subset. We use sentences as proxies for text predicates in this data. Development and test data consist of all FreeBase relations
connecting entity pairs in the development subset and 2007 data respectively. Note that entity pairs for which no FreeBase relation is indicated are not included in dev/test data, following previous literature (e.g.: (Han et al., 2018; Zhang et al., 2019)). Training data consists of remaining FreeBase relations and sentences found in the train split. Note that no development or test triples are present in the training data.
D Experimental Environment and Run Times
Experiments were performed on dual P100 GPUs with 10gb of memory in the CSIRO "Bracewell" high performance computing cluster.
Runtime per epoch with the given parameters varied by dataset and model complexity. For the REVERB extractions from ClueWeb, ENE without text and entity text enhanced models requiring approximately 10 minutes, ENE with predicate and combined entity-predicate text enhancement 15 minutes, OpenKi without text and entity text enhanced models requiring approximately 40 minutes, OpenKi with predicate and combined entity-predicate text enhancement 50 minutes.
For the NYT data, ENE without text and entity text enhanced models requiring approximately 50-60 minutes, ENE with predicate and combined entity-predicate text enhancement 60-70 minutes, OpenKi without text and entity text enhanced models requiring approximately 180 minutes, OpenKi with predicate and combined entity-predicate text enhancement 240 minutes.
E Low Dimensinal Models (from original OpenKI paper)
We ran 12 dimensional models on the NYT data to match the configuration used in the original OpenKI paper (Zhang et al., 2019). Entity neighbourhood encoding (ENE) and "dual attention" (OpenKI), without text enhancement (W/O Text) and with entity text enhancement including entity descriptions (Ent+D) — see Table 4.
These results improve on those presented in the original OpenKI paper (0.421 and 0.462 respectively) primarily due to our use of sentences as predicate proxies, which allow for a modest level of predicate co-occurrence between neighbour lists due to sentences with more than two entities. They use a window around the entity pair with the two
entities masked, resulting in no co-occurrence between neighbour lists.
Table 4: 12D NYT Models
model	W/O Text	Ent+D
ENE	0.528	0.722
OpenKI	0.558	0.581