SlowGuess/NAACL_2021_OCR huggingface dataset:
NYT-FB Hierarchy. NYT-FB (Marcheggiani and Titov, 2016) is a distantly supervised dataset created from New York Times and Freebase. Following Simon et al. (2019), we filter out sentences with non-binary relations. The train set includes 212 relations with 33,992 instances. The development set and test set share 50 relations, and have 3,835 and 3,858 instances respectively. Each relation in development set and test set has at least 10 instances. We call this dataset NYT-FB Hierarchy.
We introduce two task settings and corresponding evaluation metrics. (1) Relation clustering setting is widely adopted in previous OpenRE works to evaluate the ability of clustering novel relations (Marcheggiani and Titov, 2016; Wu et al., 2019). (2) We also design the hierarchy expansion setting to thoroughly test the ability of OpenRE models in expanding existing relation hierarchies.
4.2.1 Relation Clustering Setting
Relation clustering is a traditional OpenRE setting, where models are required to cluster instances into different groups representing new relations.
Baselines. We compare OHRE with state-of-the-art OpenRE baselines. (1) Relational Siamese Network augmented with conditional entropy and virtual adversarial training (RSN-CV) (Wu et al., 2019) is the state-of-the-art OpenRE method that transfers relational knowledge from labeled data to discover relations in unlabeled data. (2) Self-ORE (Hu et al., 2020) utilizes self-training to iteratively learn relation representations and clusters. (3) HAC with re-weighted word embeddings (RW-HAC) (Elsahar et al., 2017) is the state-of-the-art rich feature-based method. RW-HAC first extracts rich features, such as entity types, then reduces feature dimension via principal component analysis, and finally clusters the features with HAC. (4) Discrete-state variational autoencoder (VAE) (Elsahar et al., 2017) optimizes a relations classifier via reconstruction signals, with rich features including dependency paths and POS tags.
Evaluation Metrics. Following Wu et al. (2019); Hu et al. (2020), we adopt instance-level evaluation metrics to evaluate relation clustering, including  $\mathrm{B}^3$  (Bagga and Baldwin, 1998), V-measure (Rosenberg and Hirschberg, 2007) and Adjusted Rand Index (ARI) (Hubert and Arabie, 1985). We refer readers to the appendix for more detailed descriptions about the evaluation metrics.
4.2.2 Hierarchy Expansion Setting
In this setting, models are required to first cluster novel relations, and then further add the extracted relations into the existing hierarchy in train set.
Baselines. To the best of our knowledge, there are no existing OpenRE methods designed to directly expand an existing relation hierarchy. We design two strong baselines based on state-of-the-art OpenRE architectures. (1) RW-HAC for hierarchy expansion (RW-HAC-HE) links each novel relation cluster given by RW-HAC to the existing relation cluster with the global highest the Ward's linkage score. The novel relation will be a singleton if the highest score is less than a threshold. (2) RSN-CV for hierarchy expansion (RSN-CV-HE) obtains clusters using RSN-CV, and links them to the hierarchy using our top-down expansion algorithm. Here without confusion, we omit the -HE suffixes in model names in the experiment results.
Evaluation Metrics. We adopt two metrics to evaluate on cluster-level (1) how well a predicted cluster matches the golden cluster by matching metric (Larsen and Aone, 1999), and (2) how well
Dataset	Model	B3	V-measure	ARI
F1	Prec.	Rec.	F1	Hom.	Comp.
FewRel
Hierarchy	VAE (Marcheggiani and Titov, 2016)	23.0	14.2	61.4	24.1	17.7	37.9	4.9
RW-HAC (Elsahar et al., 2017)	32.7	28.0	39.4	39.7	36.0	44.4	12.4
SelfORE (Hu et al., 2020)	60.6	60.1	61.1	70.1	69.5	70.7	54.6
RSN-CV (Wu et al., 2019)	63.8	57.4	71.7	72.4	68.9	76.2	54.2
OHRE	70.5	64.5	77.7	76.7	73.8	79.9	64.2
NYT-FB
Hierarchy	VAE (Marcheggiani and Titov, 2016)	25.2	17.6	44.4	35.1	28.2	46.3	10.5
RW-HAC (Elsahar et al., 2017)	35.0	43.3	29.4	58.9	61.7	56.3	28.3
SelfORE (Hu et al., 2020)	38.1	42.6	34.5	59.0	60.7	57.5	30.4
RSN-CV (Wu et al., 2019)	38.9	26.3	74.2	44.1	74.3	55.4	26.2
OHRE	43.8	31.4	72.3	60.0	49.9	75.3	31.9
Table 1: Relation clustering results on two datasets  $(\%)$
Model	B3F1	V-F1	ARI
RSN-CV	63.8	72.4	54.2
w/o VAT	53.3	65.0	43.2
OHRE	70.5	76.7	64.2
w/o Dynamic Margin	68.9	76.1	63.5
w/o Curriculum Train	68.5	75.7	62.1
w/o Pair-wise VAT	58.3	68.8	49.5
Table 2: Ablation results on FewRel Hierarchy  $(\%)$
the predicted cluster links to the golden position in hierarchy by taxonomy metric (Dellschaft and Staab, 2006). We also report two overall evaluation metrics that consider both relation clustering and hierarchy expansion results. Specifically, we report the arithmetic mean and harmonic mean of matching F1 and taxonomy F1.
4.3 Relation Clustering Results
Main Results. Table 1 shows relation clustering results on two datasets, from which we observe that: (1) OHRE outperforms state-of-the-art models by a large margin, e.g., with  $6.7\%$ ,  $4.3\%$ ,  $9.6\%$  improvements in  $\mathbf{B}^3$ , V-measure, and ARI respectively on FewRel Hierarchy. Compared with unsupervised methods, the performance gap is even greater, e.g., more than  $30\%$  in  $\mathbf{B}^3$  on FewRel Hierarchy. This shows that OHRE can effectively leverage existing relation hierarchy for better novel relation clustering. (2) The improvements of OHRE are consistent in both supervised FewRel Hierarchy dataset and distantly supervised NYT-FB Hierarchy dataset. This indicates that the representation learning and relation clustering procedure of OHRE is robust to noisy relation labels and long-tail relations in different domains. We note that although our model adopts CNN as the relation encoder, it outperforms SelfORE equipped with BERT (Devlin et al., 2019).
We expect it would be beneficial to enhance the relation representations in OHRE with pre-trained language models, and we leave it for future work.
Ablation Study. We conduct ablations to investigate the contribution of different components, as shown in Table 2. For fair comparisons, we also ablate virtual adversarial training from RSN-CV (Wu et al., 2019). Experimental results show that all components contribute to the final performance. This shows that hierarchical information from existing relations can provide transferable guidance for novel relation clustering. The performance drops most significantly when removing pair-wise virtual adversarial training, indicating the importance of space smoothing to the generalization of OHRE.
4.4 Hierarchy Expansion Results
Main Results. Table 3 shows the results of hierarchy expansion, from which we observe that: (1) OHRE outperforms strong baselines on hierarchy expansion. Compared to baselines, OHRE achieves higher match F1, which indicates that relations extracted by OHRE can be better aligned with golden relations on cluster-level. Moreover, the advantage in taxonomy F1 shows that OHRE can better add the extracted relations in the existing hierarchy. The reasonable overall result shows the potential of OHRE in real-world open hierarchical relation extraction applications. (2) We also conduct hierarchy expansion experiments with golden novel clusters. However, experiment results show no obvious improvements for all models. Particularly, we note that while RW-HAC and RSN-CV achieve seemingly reasonable performance, they always cast novel relation as a singleton and are unable to add the relation to the right place in hierarchy.4
4The proportion of singleton relations is  $37.5\%$
(a) The expansion recommendation by OHRE on an existing relation hierarchy. The average connection score at each time step is shown.
(b) OHRE first clusters novel relations from open-domain corpora, and learns relation prototypes.
Figure 3: OHRE workflow in expanding an existing hierarchy with novel relations, and t-SNE visualization on FewRel Hierarchy. Relations with labeled instances in the dataset are marked in color. Relations in existing hierarchy are marked with solid lines, and novel relations are marked with dashed lines. Best viewed in color.
(c) OHRE then expands relation hierarchy based on relation prototypes in a top-down paradigm.
Dataset	Method	Golden Cluster	F1	Match Prec.	Rec.	Taxonomy	Arith. F1	Harm. F1
F1	Prec.	Rec.
FewRel Hierarchy	RW-HAC	33.2	33.9	37.6	37.5	37.5	37.5	35.3	35.2
RSN-CV	69.6	63.7	85.8	34.5	38.5	31.3	52.0	46.1
OHRE	78.5	73.6	88.4	53.3	57.1	50.0	65.9	63.5
RW-HAC	43.8	43.8	43.8	71.9	60.9
RSN-CV	✓	N/A	37.5	37.5	37.5	68.8	54.5
OHRE	57.4	62.5	53.1	78.7	73.0
NYT-FB Hierarchy	RW-HAC	29.6	34.3	34.0	10.1	8.7	12.0	19.8	15.0
RSN-CV	45.1	33.2	83.1	10.5	15.2	8.0	27.8	17.0
OHRE	51.7	42.7	76.2	22.3	23.9	21.0	37.0	31.2
RW-HAC	20.0	16.7	25.0	60.0	33.3
RSN-CV	✓	N/A	13.0	16.0	11.0	56.5	23.1
OHRE	23.0	23.0	23.0	61.5	37.4
Table 3: Hierarchy expansion results. Golden cluster indicates the golden relation clusters are given, in which case matching metric for relation clustering is not applicable. Arith: arithmetic mean, Harm: harmonic mean.
Relation Clustering	Hierarchy Expansion
sgl.	p-trn.	p-tst.	sgl.	p-trn.	p-tst.
RW-HAC	31.6	35.0	42.8	60.0	0.0	0.0
RSN-CV	67.1	77.8	64.4	58.8	0.0	0.0
OHRE	75.2	84.6	53.9	58.8	36.4	0.0
Table 4: Relation clustering (B $^3$  F1) and hierarchy expansion (Taxonomy F1) results on relations in different hierarchy topologies. sgl.: relations without a parent, p-trn.: parent in train set, p-tst.: parent in test set.
This is because the inconsistent instance representations within each golden cluster will mislead the expansion procedure on cluster-level, which shows integrating hierarchy information into relation representations is of fundamental importance to hierarchy expansion. Besides, the results also show the necessity of re-splitting FewRel to include more hierarchy topologies in test set for better benchmark.
Zoom-in Study. To better understand the performance of models on hierarchy expansion, we divide
the relations according to their hierarchy topologies and report the performance on FewRel Hierarchy. Table 4 shows the results on three topologies, including (1) single relations without parents (tgl.), (2) relations with parents in train set (p-trn.), and (3) relations with parents in test set (p-tst.). The results show that although models achieve reasonable performance on clustering in all three topologies, they struggle on hierarchy expansion, especially on relations with parents. In comparison, OHRE can handle some relations with parents in train set. However, there is still ample room for improvement. This shows hierarchy expansion is challenging, and we leave further research for future work.
To intuitively show how OHRE expands an existing hierarchy with novel relations from open-domain corpora, we visualize the workflow of OHRE on relation composer, as shown in Figure 3. The average connection score increases as the expansion proce
dure progress from top to down in hierarchy. The expansion procedure terminates when the connection score decreases. The process is not only better aligned with real-world needs, but also provides better interpretability in decision making.
In this work, we make the first attempt to address bidirectional connections between OpenRE and relation hierarchy. In the future, we believe the following directions worth exploring: (1) We use a heuristic method to add new relations into hierarchies based on local similarities between relations. In future, more advanced methods can be designed to model the global interaction between new relations and hierarchy, and learn to effectively add the novel relations. (2) We conduct relation representation learning and hierarchy expansion in a pipeline. In the future, end-to-end models can be developed to jointly optimize these important phases for better open hierarchical relation extraction results.
This work is supported by the National Key Research and Development Program of China (No. 2020AAA0106501). Yao is also supported by 2020 Tencent Rhino-Bird Elite Training Program.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of ACL-IJCNLP, pages 344-354. ACL.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722-735. Springer.
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In Proceedings of ACL-COLING, pages 79-85. ACL.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of IJCAI, pages 2670-2676. ACM.
Michele Banko and Oren Etzioni. 2008. The tradeoffs between open and traditional relation extraction. In Proceedings of ACL: HLT, pages 28-36. ACL.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of ICML, page 41-48. ACM.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250.
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proceedings of ACL, pages 407-413. ACL.
Klaas Delfschaft and Steffen Staab. 2006. On how to perform a gold standard based evaluation of ontology learning. In Proceedings of ISWC, pages 228-241. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL: HLT, pages 4171-4186. ACL.
Hady Elsahar, Elena Demidova, Simon Gottschalk, Christophe Gravier, and Frederique Laforest. 2017. Unsupervised open relation extraction. In Proceedings of ESWC, pages 12-16. Springer.
Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. 2018a. Hierarchical relation extraction with coarse-to-fine grained attention. In Proceedings of EMNLP, pages 2236-2245. ACL.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018b. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP, pages 4803-4809. ACL.
Xuming Hu, Lijie Wen, Yusong Xu, Chenwei Zhang, and Philip Yu. 2020. SelfORE: Self-supervised relational feature learning for open relation extraction. In Proceedings of EMNLP, pages 3673-3682. ACL.
Zhiting Hu, Poyao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing. 2015. Entity hierarchy embedding. In Proceedings of ACL-IJCNLP, pages 1292-1300. ACL.
Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 2(1):193-218.
Bjornar Larsen and Chinatsu Aone. 1999. Fast and effective text mining using linear-time document clustering. In Proceedings of KDD, page 16-22, New York, NY, USA. ACM.
Claudia Leacock and Martin Chodorow. 1998. Combining local context and wordnet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265-283.
Dekang Lin and Patrick Pantel. 2001. Dirt @sbt@discovery of inference rules from text. In Proceedings of KDD. ACM Press.
Diego Marcheggiani and Ivan Titov. 2016. Discrete-state variational autoencoders for joint discovery and factorization of relations. TACL, 4:231-244.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In Proceedings of ICLR.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543. ACL.
Simone Paolo Ponzetto and Michael Strube. 2007. Knowledge derived from wikipedia for computing semantic relatedness. JAIR, 30:181-212.
Andrew Rosenberg and Julia Hirschberg. 2007. Vmeasure: A conditional entropy-based external cluster evaluation measure. In Proceedings of EMNLP-ConL, pages 410-420. ACL.
Juho Rousu, Craig Saunders, Sandor Szedmak, and John Shawe-Taylor. 2005. Learning hierarchical multi-category text classification models. In Proceedings of ICML, pages 744-751. ACM.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of CVPR, pages 815-823. IEEE Computer Society.
Étienne Simon, Vincent Guigue, and Benjamin Piwowarski. 2019. Unsupervised information extraction: Regularizing discriminative approaches with relation distribution losses. In Proceedings of ACL, pages 1378-1387. ACL.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(56):1929-1958.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of NAACL: HLT, pages 885-895. ACL.
Joshua B. Tenenbaum, Charles Kemp, Thomas L. Grifths, and Noah D. Goodman. 2011. How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022):1279-1285.
Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57:78-85.
Kilian Q Weinberger and Olivier Chapelle. 2009. Large margin taxonomy embedding for document categorization. In Advances in NeurIPS, pages 1737-1744. Curran Associates, Inc.
Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data. In Proceedings of EMNLP-IJCNLP, pages 219-228. ACL.
Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016. Representation learning of knowledge graphs with hierarchical types. In Proceedings of IJCAI, pages 2965-2971. IJCAI/AAAI Press.
Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011. Structured relation discovery using generative models. In Proceedings of EMNLP, pages 1456-1466. ACL.
Limin Yao, Sebastian Riedel, and Andrew McCallum. 2012. Unsupervised relation discovery with sense disambiguation. In Proceedings of ACL, pages 712-720. ACL.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 2335-2344. ACL.
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. In Proceedings of NAACL: HLT, pages 3016-3025. ACL.
In this section, we introduce hyperparameters and important bounds in relation representation learning and in relation hierarchy expansion respectively. All hyperparameters are selected by grid search on the development set. Moreover, we report the average training time and the number of parameters.
Representation Learning Hyperparameters. In embedding layer, we use 50-d GloVe (Pennington et al., 2014) word embeddings and 2 randomly initialized 5-d position embeddings, and all the embeddings are trainable. The convolution kernel size is 3, relation embedding size is 64 selected from  $\{64,128,256,512\}$ , and  $\lambda_{d}$  in representation learning is 0.7 selected from  $\{0.4,0.5,0.6,0.7,0.8,0.9,1\}$ . We apply dropout (Srivastava et al., 2014) after embedding layer with dropout rate 0.2, and L2 regularization on the convolutional and fully connected layer with hyperparameters 2e-4 and 1e-3 respectively. During training, the batch size is 64 selected from  $\{16,32,64,128\}$ . For each batch, we randomly sample 4 relation types, each with 16 instances. Besides, hierarchical curriculum learning strategy lasts 100 batches in the first epoch to warm up the model parameters. In pair-wise virtual adversarial training strategy, we first generate perturbation vector  $\delta_{1}$  for each instance representation  $\mathbf{v}$ , where the value in each dimension follows a uniform distribution of range  $[0,1)$ . Then the perturbation vector  $\delta_{1}$  is scaled such that its L2 norm is 0.02. We add  $\delta_{1}$  to the instance feature, and compute the worst-case perturbation  $\delta_{2}$  based on the gradient. Finally  $\delta_{2}$  is scaled to 0.02 in L2 norm, and added to the feature of the instance to obtain  $\tilde{\mathbf{v}}$ .
Hierarchy Expansion Hyperparameters. In relation clustering process, Louvain (Blondel et al., 2008) algorithm will not take the similarity between instances less than the threshold 0.5 into account. The instance of novel relation prototypes having less than 5 instances will be moved to their closest neighbors based on the average connection weight. During hierarchy expansion, the thresholds for singleton relations in top-down expansion and RW-HAC-HE are 0.2 and 0.1 respectively.
B Evaluation Metrics
In this section, we provide details of evaluation metrics in two settings.
Relation Clustering Setting. Following previous
works (Wu et al., 2019; Hu et al., 2020), we adopt instance-level evaluation metrics, including  $\mathrm{B}^3$ , V-measure and Adjusted Rand Index.
(1)  $\mathrm{B}^3$ . For each instance in test set,  $\mathrm{B}^3$  computes its precision and recall by comparing the cluster containing the instance in prediction results and the cluster containing the instance in golden answer. After that,  $\mathrm{B}^3$  averages the precision and recall of each instance and produces a harmonic mean. (2) V-measure. Similarly, V-measure (Rosenberg and Hirschberg, 2007) is another instance-based measurement that further introduces conditional entropy, which asks for the higher requirement of the purity of clusters. Compare to  $\mathrm{B}^3$ , the existence of a few wrong instances in a relatively pure cluster decreases more score to punish clustering results. Meanwhile, the V-measure F1 calculates the harmonic mean of homogeneity and completeness. (3) Adjusted Rand Index. ARI (Hubert and Arabie, 1985) counts all pair-wise assignments in the same or different groups to measure the similarity of predicted and golden clusterings. Random node assignment makes ARI be 0, and the maximum of ARI is 1, which means the perfect result. Compared to the previous two metrics, ARI is less sensitive since it won't be influenced by an extreme sub-value like precision or homogeneity.
Hierarchy Expansion Setting. To bridge the predicted clusters with real relations, we first match each predicted cluster to the golden cluster then cast it as a prototype for hierarchy position evaluation. We borrow two metrics to evaluate how well a predicted cluster matches the golden cluster, and how well the predicted cluster links to the golden position in hierarchy on cluster-level.
(1) Matching Metric. Similar to Larsen and Aone (1999), we try to match each predicted cluster to one golden relation with whom the predicted cluster has the highest F1 score on cluster-level. Note that different from the original measurement, the golden relation can be matched once only. For each paired novel cluster and golden relation, we calculate precision, recall, and F1 score, and finally weighted sum up based on the number of instances.
(2) Taxonomy Metric. Taxonomy metric was first proposed to evaluate taxonomy structure (Dellschaft and Staab, 2006). After matching predicted clusters to golden relations, for each predicted cluster, we use taxonomy metric to compare the position of this predicted cluster and the position of the corresponding golden relation in
hierarchy. Assume position  $p$  in hierarchy is characterized by the union of all its ancestors and descendants  $u(p)$ . Denote  $r_g$  as the golden position and  $r_p$  as the predicted position of relation  $r$  in the hierarchy, respectively. The precision is defined as follows,
$$
\operatorname {P r e c.} = \frac {1}{| P |} \sum_ {r \in P} \frac {\left| u \left(r _ {p}\right) \cap u \left(r _ {g}\right) \right|}{\left| u \left(r _ {p}\right) \right|}, \tag {8}
$$