Abstract: We introduce a conceptually simple and effective method to quantify the similarity between relations in knowledge bases. Specifically, our approach is based on the divergence between the conditional probability distributions over entity pairs. In this paper, these distributions are parameterized by a very simple neural network. Although computing the exact similarity is in-tractable, we provide a sampling-based method to get a good approximation. We empirically show the outputs of our approach significantly correlate with human judgments. By applying our method to various tasks, we also find that (1) our approach could effectively detect redundant relations extracted by open information extraction (Open IE) models, that (2) even the most competitive models for relational classification still make mistakes among very similar relations, and that (3) our approach could be incorporated into negative sampling and softmax classification to alleviate these mistakes. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/relation-similarity.

Introduction: Author contributions: Hao Zhu designed the research; Weize Chen prepared the data, and organized data annotation; Hao Zhu and Xu Han designed the experiments; Weize Chen performed the experiments; Hao Zhu, Weize Chen and Xu Han wrote the paper; Zhiyuan Liu and Maosong Sun proofread the paper. Zhiyuan Liu is the corresponding author. Relations, representing various types of connections between entities or arguments, are the core of expressing relational facts in most general knowledge bases (KBs) BIBREF0 , BIBREF1 . Hence, identifying relations is a crucial problem for several information extraction tasks. Although considerable effort has been devoted to these tasks, some nuances between similar relations are still overlooked, (tab:similarityexample shows an example); on the other hand, some distinct surface forms carrying the same relational semantics are mistaken as different relations. These severe problems motivate us to quantify the similarity between relations in a more effective and robust method. In this paper, we introduce an adaptive and general framework for measuring similarity of the pairs of relations. Suppose for each relation INLINEFORM0 , we have obtained a conditional distribution, INLINEFORM1 ( INLINEFORM2 are head and tail entities, and INLINEFORM3 is a relation), over all head-tail entity pairs given INLINEFORM4 . We could quantify similarity between a pair of relations by the divergence between the conditional probability distributions given these relations. In this paper, this conditional probability is given by a simple feed-forward neural network, which can capture the dependencies between entities conditioned on specific relations. Despite its simplicity, the proposed network is expected to cover various facts, even if the facts are not used for training, owing to the good generalizability of neural networks. For example, our network will assign a fact a higher probability if it is “logical”: e.g., the network might prefer an athlete has the same nationality as same as his/her national team rather than other nations. Intuitively, two similar relations should have similar conditional distributions over head-tail entity pairs INLINEFORM0 , e.g., the entity pairs associated with be trade to and play for are most likely to be athletes and their clubs, whereas those associated with live in are often people and locations. In this paper, we evaluate the similarity between relations based on their conditional distributions over entity pairs. Specifically, we adopt Kullback–Leibler (KL) divergence of both directions as the metric. However, computing exact KL requires iterating over the whole entity pair space INLINEFORM1 , which is quite intractable. Therefore, we further provide a sampling-based method to approximate the similarity score over the entity pair space for computational efficiency. Besides developing a framework for assessing the similarity between relations, our second contribution is that we have done a survey of applications. We present experiments and analysis aimed at answering five questions: (1) How well does the computed similarity score correlate with human judgment about the similarity between relations? How does our approach compare to other possible approaches based on other kinds of relation embeddings to define a similarity? (sec:relationship and sec:human-judgment) (2) Open IE models inevitably extract many redundant relations. How can our approach help reduce such redundancy? (sec:openie) (3) To which extent, quantitatively, does best relational classification models make errors among similar relations? (sec:error-analysis) (4) Could similarity be used in a heuristic method to enhance negative sampling for relation prediction? (sec:training-guidance-relation-prediction) (5) Could similarity be used as an adaptive margin in softmax-margin training method for relation extraction? (sec:training-guidance-relation-extraction) Finally, we conclude with a discussion of valid extensions to our method and other possible applications.

Question: Which competitive relational classification models do they test?
Answer: For relation prediction they test TransE and for relation extraction they test position aware neural sequence model