SlowGuess/ACL_2020_OCR huggingface dataset:
pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1116, Berlin, Germany. Association for Computational Linguistics.
Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 39-48, Denver, Colorado. Association for Computational Linguistics.
Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In Proceedings of the 26th International Conference on World Wide Web, pages 1015-1024.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pages 148-163. Springer.
Simone Romano, Nguyen Xuan Vinh, James Bailey, and Karin Verspoor. 2016. Adjusting for chance clustering comparison measures. Journal of Machine Learning Research, 17(1):4635-4666.
Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 410-420, Prague, Czech Republic. Association for Computational Linguistics.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Étienne Simon, Vincent Guigue, and Benjamin Piwowarski. 2019. Unsupervised information extraction: Regularizing discriminative approaches with relation distribution losses. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1378-1387, Florence, Italy. Association for Computational Linguistics.
Kiri Wagstaff. 2000. Refining inductive bias in unsupervised learning via constraints. In AAAI/IAAI, page 1112.
Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011. Structured relation discovery using generative models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1456-1466, Edinburgh,
7503
Scotland, UK. Association for Computational Linguistics.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205-2215, Brussels, Belgium. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Table 3 shows the statistics of the NYT-FB (Marcheggiani and Titov, 2016) and TA-CRED (Zhang et al., 2017) datasets. We followed the same data split and pre-processing described in Marcheggiani and Titov (2016). For all methods, we trained on NYT-FB and evaluated them on both NYT-FB and TACRED.
Figure 2 illustrates the relation distributions of two datasets: NYT-FB and TACRED. We can see that 15/253 most frequent relations account for  $82.97\%$  of the total number of instances in NYT-FB. Meanwhile, 15/41 relations sum up to  $74.94\%$  of the total number of instances in TACRED.
B Hyper-parameter Settings
We used the development set to stop the training process. For every model, we conducted three runs with different initialised parameters and computed the average performance. We list the hyperparameters of different models in Table 4.
C Detailed Results
Table 5 presents the average test scores of three runs on the NYT-FB and TACRED datasets. We note that the two models proposed by Marcheggiani and Titov (2016) and Simon et al. (2019) are sensitive to the hyper-parameters and thus difficult to train. We could not replicate the performance of Simon on the NYT-FB dataset.
Train	Dev	Test
NYT-FB (#r = 262)
Raw instances	1,950,557	389,819	1,560,738
Positive	41,685	7,793	33,808
TACRED (#r = 41)
Raw instances	68,124	22,631	15,509
Positive	13,012	5,436	3,325
Table 3: The statistics of the NYT-FB and the TACRED datasets.  $\# r$  indicates the number of relation types in each dataset.
Parameter	Ls	Ls + Ld
Optimiser	AdaGrad
Number of epochs	10
Batch size	100
L2 regularisation	1e-7
Feature dimension	10
Learning rate	0.1	0.005
Ls coefficient	0.1	0.01
Ld coefficient	-	0.02
(a) Marcheggiani and Titov (2016)'s model.
Parameter	Value
Optimiser	Adam
Learning rate	0.005
Learning rate annealing	0.50.25
Batch size	100
Early stop patience	10
L2 regularisation	2e-11
Word dimension	50
Entity type dimension	10
Ls coefficient	0.01
Ld coefficient	0.02
(b) Simon et al. (2019)'s model.
Parameter	Value
Optimiser	Adam
Learning rate	0.001
Batch size	100
Early stop patience	10
L2 regularisation	1e-5
Entity type dimension	10
Ls coefficient	0.0001
Ld coefficient	0.02
(c) EType+.
Table 4: Hyper-parameter values used in our experiments.
Model	F1	B3P	R	F1	V-measure Hom.	Comp.	ARI
NYT-FB
RelLDA	29.1	24.8	35.2	30.0	26.1	35.1	13.3
RelLDA1	36.9	30.4	47.0	37.4	31.9	45.1	24.2
March (Ls+Ld)	n = 10	37.5	31.1	47.4	38.7	32.6	47.8	27.6
March (Ls+Ld)‡	38.7	30.9	51.7	37.6	31.0	47.7	26.1
Simon	39.4	32.2	50.7	38.3	32.2	47.2	33.8
Simon‡	32.6	28.2	38.9	30.5	26.1	36.8	23.8
ETYPE+	41.9	31.3	63.7	40.6	31.8	56.2	30.7
March (Ls+Ld)‡	n = 16	36.9	32.0	43.7	37.4	32.6	43.9	28.1
ETYPE	41.7	32.5	58.0	42.1	34.7	53.6	30.7
ETYPE+	41.5	32.0	59.0	41.3	33.6	53.9	30.5
RelLDA1	n = 100	29.6	-	-	-	-	-	-
March	35.8	-	-	-	-	-	-
March‡	34.8	24.4	62.4	25.9	18.7	42.7	13.1
TACRED
March (Ls+Ld)‡	n = 10	31.0	21.7	54.9	43.8	35.5	57.2	22.6
Simon‡	15.7	12.1	22.4	17.1	14.6	20.6	6.1
ETYPE+	43.3	28.0	96.9	59.7	43.4	96.0	25.7
March (Ls+Ld)‡	n = 16	34.6	24.3	61.3	47.6	38.9	61.4	23.2
ETYPE	48.3	32.3	96.3	64.4	48.6	95.6	29.1
ETYPE+	46.1	30.3	96.9	62.0	45.8	96.1	27.4
March‡	n = 100	33.13	21.83	69.20	43.63	32.96	64.66	20.21
Table 5: Average results (\%) across three runs of different models (except the rule-based EType) on two datasets: the distant supervision NYT-FB and the large supervised dataset TACRED. The model of Marcheggiani and Titov (2016) is March and the model of Simon et al. (2019) is Simon.  $\ddagger$  indicates our implementation of the corresponding model.
(a) NYT-FB has 253 relation types in total
Figure 2: Relation distribution of NYT-FB and TACRED  $(\%)$ .
(b) TACRED has 41 relation types in total