Venue: ACL 2020

# Connecting Embeddings for Knowledge Graph Entity Typing

Yu Zhao1, Anxiang Zhang2, Ruobing Xie3, Kang Liu45, Xiaojie Wang6

Equal Contribution. Corresponding author: Y. Zhao (zhaoyu@swufe.edu.cn).

###### Abstract

Knowledge graph (KG) entity typing aims at inferring possible missing entity type instances in KG, which is a very significant but still under-explored subtask of knowledge graph completion. In this paper, we propose a novel approach for KG entity typing which is trained by jointly utilizing _local typing knowledge_ from existing entity type assertions and _global triple knowledge_ from KGs. Specifically, we present two distinct knowledge-driven effective mechanisms of entity type inference. Accordingly, we build two novel embedding models to realize the mechanisms. Afterward, a joint model with them is used to infer missing entity type instances, which favors inferences that agree with both entity type instances and triple knowledge in KGs. Experimental results on two real-world datasets (Freebase and YAGO) demonstrate the effectiveness of our proposed mechanisms and models for improving KG entity typing. The source code and data of this paper can be obtained from: [https://github.com/Adam1679/ConnectE](https://github.com/Adam1679/ConnectE)

## 1 Introduction

The past decade has witnessed great thrive in building web-scale knowledge graphs (KGs), such as Freebase Bollacker et al. (2008), YAGO Suchanek et al. (2007), Google Knowledge Graph Dong et al. (2014), which usually consists of a huge amount of triples in the form of (_head entity_, _relation_, _tail entity_) (denoted \((e,r,\hat{e})\)). KGs usually suffer from _incompleteness_ and miss important facts, jeopardizing their usefulness in downstream tasks such as question answering Elsahar et al. (2018), semantic parsing Berant et al. (2013), relation classification Zeng et al. (2014). Hence, the task of knowledge graph completion (KGC, i.e. completing knowledge graph entries) is extremely significant and attracts wide attention.

This paper concentrates on KG entity typing, i.e. inferring missing entity type instances in KGs, which is an important sub-problem of KGC. Entity type instances, each of which is in the formed of (_entity, entity type_) (denoted \((e,t)\)), are essential entries of KGs and widely used in many NLP tasks such as relation extraction Zhang et al. (2018); Jain et al. (2018), coreference resolution Hajishirzi et al. (2013), entity linking Gupta et al. (2017). Most previous works of KGC focus on inferring missing entities and relationships Bordes et al. (2013); Wang et al. (2014); Lin et al. (2015); Dettmers et al. (2017); Ding et al. (2018); Nathani et al. (2019), paying less attention to entity type prediction. However, KGs also usually suffer from entity types incompleteness. For instance, 10% of entities in _FB15k_Bordes et al. (2013), which have the _/music/atrist_ type, miss the _/people/person_ type Moon et al. (2017). KG entity type incompleteness leads to some type-involved algorithms in KG-driven tasks grossly inefficient or even unavailable.

To solve KG entity type incompleteness issue, in this paper we propose a novel embedding methodology to infer missing entity type instances that employs not only _local typing knowledge_ from entity type assertions, as most conventional mod

Figure 1: Effective mechanisms of entity type inference with local typing knowledge and global triple knowledge.



els do, but also leverages _global triple knowledge_ from KGs. Accordingly, we build two distinct knowledge-driven type inference mechanisms with these two kinds of structural knowledge.

**Mechanism 1**.: _Missing entity types of an entity can be found from other entities that are close to the entity in the embedding space, using local typing knowledge as in Fig.\(1\)(Mech.1)._

**Mechanism 2**.: _Missing entity types of an (head or tail) entity can be inferred from the types of other (tail or head) entities through their relationships, using global triple knowledge as in Fig.\(1\)(Mech.2)._

The main idea behind Mech.\(1\) is based on the observation that the learned entities' embeddings by conventional KG embedding methods Ji et al. (2016); Xie et al. (2016) cluster well according to their types in vector space. For instance, in Fig.\(1\)(Mech.1), given an entity _Barack Obama_, it's missing hierarchical type /_people/person_ can be induced by the given hierarchical type of similar entity _Donald Trump_. In addition, the key motivation behind Mech.\(2\) is that the relationship shall remain unchanged if the entities in a triple fact are replaced with their corresponding hierarchical types. For instance, given a global triple fact (_Barack Obama_, _born_in, Honolulu_), under this assumption, we can induce a new type triple (_/people/person, born_in, /location/location_)1. Formally, _Honolulu -- Barack Obama = location/location - /people/person_ (= _born_in_), which can be used to infer missing entity types, e.g. (_Barack Obama_, _type_=?_ ) via _Barack Obama - Honolulu + Iocation/location = /people/person_, as Mech.2 does. Fig.\(1\) demonstrates a simple illustration of effective mechanisms of entity type inference. Both mechanisms are utilized to build our final composite model.

Footnote 1: For more clarity, we represent it as (_Iocation/location, born_in^-1_, /people/person_) in Fig.\(1\)(Mech.2).

Specifically, we build two embedding models to realize the two mechanisms respectively. First, considering entities and entity types are completely distinct objects, we build two distinct embedding spaces for them, i.e., **entity space** and **entity type space**. Accordingly, we encode \((e,t)\) entity type instance by projecting the entity from entity space to entity type space with mapping matrix \({\bf M}\), hence we have (1): \(\left|\begin{array}{c}{\bf M}\cdot{\bf e}\simeq{\bf t}\\ \end{array}\right|\), called **E2T**. Moreover, we learn the plausibility of (\(t_{e},r,t_{\tilde{e}}\)) global type triple by newly generalizing from (\(e,r,\tilde{e}\)) global triple fact, even though this type triple is not present originally. Following translating assumption Bordes et al. (2013), we have (2): \(\left|\begin{array}{c}{\bf f}_{\tilde{e}}-{\bf r}^{\circ}\simeq{\bf t}_{e} \\ \end{array}\right|\), called **TRT**. E2T and TRT are the implementation models of the two mechanisms. Fig.\(2\) demonstrates a brief illustration of our models. A ranking-based embedding framework is used to train our models. Thereby, entities, entity hierarchical types, and relationships are all embedded into low-dimensional vector spaces, where the composite energy score of both E2T and TRT are computed and utilized to determine the optimal types for (_entity, entity type_=?) incomplete assertions. The experimental results on real-world datasets show that our composite model achieves significant and consistent improvement compared to all baselines in entity type prediction and achieves comparable performance in entity type classification.

**Our contributions** are as follows:

* We propose a novel framework for inferring missing entity type instances in KGs by connecting entity type instances and global triple information and correspondingly present two effective mechanisms.
* Under these mechanisms, we propose two novel embedding-based models: one for predicting entity types given entities and another one to encode the interactions among entity types and relationships from KGs. A combination of both models are utilized to conduct entity type inference.
* We conduct empirical experiments on two real-world datasets for entity type inference, which demonstrate our model can successfully take into account global triple information to improve KG entity typing.

## 2 Related Works

Entity typing is valuable for many NLP tasks Yaghoobzadeh et al. (2018), such as knowledge base population Zhou et al. (2018), question answering Elsahar et al. (2018), etc. In recent years, researchers attempt to mine fine-grained entity types Yogatama et al. (2015); Choi et al. (2018); Xu and Barbosa (2018); Yuan and Downey (2018) with external text information, such as web search query logs Pantel et al. (2012), the textual surface patterns Yao et al. (2013), context representation Abhishek et al. (2017), Wikipedia Zhou et al. ( 

2018). Despite their success, existing methods rely on additional external sources, which might not be feasible for some KGs.

To be more universal, Neelakantan et al. (2015) propose two embedding models, i.e. linear model (LM) and projection embedding model (PEM), which can infer missing entity types only with KG itself. Although PEM has more expressive power than LM, however, both of them ignore global triple knowledge, which could also be helpful for encoding entity type assertions via shared entities' embeddings. To address this issue, Moon et al. (2017) propose a state-of-the-art model (ETE) to combine triple knowledge and entity type instances for entity type prediction, and build two entity type embedding methodologies: (1) Synchronous training: treat _(entity, entity type)_ assertions as special triple facts that have a unique relationship "_rdf:type_", e.g. _(Barack Obama, "rdf:type", person)_, and encode all mixed triple facts (original triple data fused with all generated special ones) by conventional entity relation embedding models, such as RESCAL Nickel et al. (2011), HOLE Nickel et al. (2016) and TransE Bordes et al. (2013). (2) Asynchronous training: first learn the entities' embeddings \({\bf e}\) by conventional entity relation embedding models mentioned above, and then only update entity types' embeddings \({\bf t}\) for \(\min\|{\bf e}-{\bf t}\|_{\ell 1}\) while keeping \({\bf e}\) fixed, called RESCAL-ET, HOLE-ET, TransE-ET and ETE. Although these approaches expect to explore global triple knowledge for entity type prediction, they still lack of expressive ability due to its simplicity of embeddings. In addition, they irrationally assume both the embeddings of entities and entity types being in the same latent space (\(\in\mathbb{R}^{n}\)). Since entities and entity types are completely distinct objects, it may not be reasonable to represent them in a common semantic space.

In this paper, we introduce an enhanced KG entity type embedding model with better expressing and reasoning capability considering both local entity typing information and global triple knowledge in KGs. Note that incorporating more external information Jin et al. (2018); Neelakantan et al. (2015) is not the main focus in this paper, as we only consider the internal structural information in KGs instead, which correspondingly makes our work much more challenging but also more universal and flexible due to the limited information. Recently, Lv et al. (2018); Hao et al. (2019) also attempt to embedding structural information in KG. However, the goals and models are very different from ours. They encodes the concepts, not hierarchical types. On the contrary, we focus on the latter not the former. Table 1 summarizes the energy functions and other different settings of entity type embedding models.

## 3 Embedding-based Framework

We consider a KG containing entity type instances of the form \((e,t)\in\mathcal{H}\) (\(\mathcal{H}\) is the training set consists of lots of (_entity, entity type_) assertions), where \(e\in\mathcal{E}\) (\(\mathcal{E}\) is the set of all entities) is an entity in the KG with the type \(t\in\mathcal{T}\) (\(\mathcal{T}\) is the set of all types). For example, \(e\) could be _Barack Obama_ and \(t\) could be _/people/person_. As a single entity can have multiple types, entities in KG often miss some of their types. The aim of this work is to infer missing entity type instances in KGs.

Our work concerns energy-based methods,

\begin{table}
\begin{tabular}{l||c|c|c|c|c} \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c|}{**Energy function**} & \multirow{2}{*}{**Parameters**} & \multirow{2}{*}{**Sources**} & **Training strategy** \\ \cline{2-4}  & \({\bf S}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{0}\texttt{s}\boldsymbol{z}}}({ \boldsymbol{e}},{\boldsymbol{t}})\) & \({\bf S}_{triple}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{0}\texttt{s}})\) & & \\ \hline LMNeelakantan et al. (2015) & \({\bf e}^{\top}{\bf t}\) & N/A & \({\bf e},{\bf t}\in\mathbb{R}^{c}\) & entity type instances & N/A \\ \hline PEMNeelakantan et al. (2015) & \({\bf e}^{\top}{\bf U}{\bf V}^{\top}{\bf t}\) & N/A & \(\begin{array}{c}{\bf e}\in\mathbb{R}^{c},{\bf t}\in\mathbb{R}^{d},\\ {\bf U}\in\mathbb{R}^{m\times d},{\bf V}\in\mathbb{R}^{k\times d}\end{array}\) & entity type instance & N/A \\ \hline RESCAL Nickel et al. (2011) & N/A & \({\bf e}^{\top}{\bf M}_{r}\tilde{\boldsymbol{e}}\) & \({\boldsymbol{e}},\tilde{\boldsymbol{e}}\in\mathbb{R}^{n},{\bf M}_{r}\in\mathbb{ R}^{c\times n}\) & mixed tuple knowledge & syn. \\ \hline RESCAL-ET Moon et al. (2017) & \(\|{\bf e}-{\bf t}\|_{1}\) & \({\bf e}^{\top}{\bf M}_{r}\tilde{\boldsymbol{e}}\) & \({\boldsymbol{e}},\tilde{\boldsymbol{e}},{\boldsymbol{t}}\in\mathbb{R}^{n},{\bf M} _{r}\in\mathbb{R}^{n\times k}\) & entity type inst./ triple know. & aspyn. \\ \hline HOLE Nickel et al. (2016) & N/A & \({\bf r}^{\top}({\bf e}\star\tilde{\boldsymbol{e}})\) & \({\bf e},{\bf r},\tilde{\boldsymbol{e}}\in\mathbb{R}^{n}\) & mixed tuple knowledge & syn. \\ \hline HOLE-ET Moon et al. (2017) & \(\|{\bf e}-{\bf t}\|_{1}\) & \({\bf r}^{\top}({\bf e}\star\tilde{\boldsymbol{e}})\) & \({\bf e},{\bf r},\tilde{\boldsymbol{e}},{\boldsymbol{t}}\in\mathbb{R}^{n}\) & entity type inst./ triple know. & aspyn. \\ \hline TranBers de Berges et al. (2013) & N/A & \(\|{\bf e}+{\mathbf{r}}-\bar{\boldsymbol{e}}\|\) & \({\bf e},{\bf r},\tilde{\boldsymbol{e}}\in\mathbb{R}^{n}\) & mixed tuple knowledge & syn. \\ \hline TransE-ET Moon et al. (2017) & \(\|{\bf e}-{\bf t}\|_{1}\) & \(\|{\bf e}+{\mathbf{r}}-\bar{\boldsymbol{e}}\|\) & \({\bf e},{\bf r},\tilde{\boldsymbol{e}},{\boldsymbol{t}}\in\mathbb{R}^{n}\) & entity type inst./ triple know. & aspyn. \\ \hline ETE Moon et al. (2017) & \(\|{\bf e}-{\bf t}\|_{1}\) & \(\|{\bf e}+\bar{\boldsymbol{e}}+{\bf c}-{\bf r}\|\) & \({\bf e},{\bf r},\tilde{\boldsymbol{e}},{\mathbf{t}}\in\mathbb{R}^{n}\) & entity type inst./ triple know. & aspyn. \\ \hline \multirow{2}{*}{**ConnectE (our proposed)**} & \(\|{\bf M}\cdot{\bf e}-{\bf t}\|_{2}^{2}\) & \(\begin{array}{c}{\bf e}^{\top}{\bf r}^{\top}-\bar{\boldsymbol{e}}\|_{2}^{2},\\ {\bf t}_{\bf c}+{\bf r}^{\top}-{\bf t}\|_{2}^{2}\end{array}\) & \({\bf e},{\bf r}^{\top}\in\mathbb{R}^{c},{\bf t},{\bf r}^{\top}\in\mathbb{R}^{d}\) & entity type inst./ triple know. & spyn. \\ \hline \end{tabular}
\end{table}
Table 1: Entity type embedding models.

 

which learn low-dimensional vector representations (_embeddings_) of atomic symbols (i.e. _entities, entity hierarchical types, relationships_). In this framework, we learn two submodels: (1) one for predicting entity types given entities, and (2) another one to encode the interactions among entity types and relationships from KGs. The joint action of both models in prediction allows us to use the connection between triple knowledge and entity type instances to perform KG entity typing.

### E2T: Mapping Entities to Types

The first model (E2T) of the framework concerns the learning of a function \({\bf S}_{e2t}(e,t)\) with local typing knowledge from entity type instances, which is designed to score the similarity of an entity \(e\) and a type \(t\). The main ideas behind this model are as follows: (1) Since the learned entity embeddings cluster well when they have the same or similar types, therefore, it is rather intuitive that the entity type embedding represents the projective common concept representation of a cluster of entities, i.e., \(f_{proj}({\bf e})\simeq{\bf t}_{e},\forall e\in{\cal E}\). \({\bf e}~{}(\in\mathbb{R}^{\ell})\) is the embedding of the entity \(e\), \({\bf t}_{e}~{}(\in\mathbb{R}^{\ell})\) is the embedding of the type \(t_{e}\). The entity type embedding represents common information of their entities, it thus should have fewer variates, i.e., \(\ell<\kappa\). (2) Since the entities and entity types are totally distinct objects, we respectively build two embedding space for them, i.e., **entity space** and **entity type space**. (3) Inspired by the previous work TranSparse Ji et al. (2016) projecting entities from entity space to relation space with operation matrix \({\bf M}\), which we adapted, replacing relation space with entity type space, we thus define \(f_{proj}({\bf e})={\bf M}\cdot{\bf e}~{}(\simeq{\bf t}_{e})\). Therefore, this model consists of first projecting entity embedding into entity type space, and then computing a similarity measure between this projection and an entity type embedding. The scoring function of E2T given (\(e,t\)) is:

\[{\bf S}_{e2t}(e,t)=\|{\bf M}\cdot{\bf e}-{\bf t}\|_{\ell 2}^{2}~{}, \tag{1}\]

where \({\bf M}\in\mathbb{R}^{\ell\times\kappa}\) is a transfer matrix mapping entity embeddings into entity type space. The score is expected to be lower for a golden entity type instance and higher for an incorrect one.

### TRT: Encoding Triples in KGs

Using only entity type instances for training ignores much of relational knowledge that can leverage from triple facts in KGs. In order to connect this relational data with our model, we propose to learn entity type and relationship embeddings from global triple knowledge from KGs. The key motivations behind this model are: (1) As mentioned above, the entities cluster well according to their types. Therefore, we believe that an essential premise of a triple (_head entity, relationship, tail entity_) holds is that its corresponding entity types should first conform to this relationship. Hence, we can build a new **entity type triple** (_head type, relationship, tail type_) by replacing both head entity and tail entity with their corresponding types: i.e. \((e,r,\tilde{e})\xrightarrow{replace}(t_{e},r,t_{\tilde{e}})\). \((e,r,\tilde{e})\in{\cal D}\), \({\cal D}\) is the training set consists of a lot of triples. \(r\in{\cal R}\) (\({\cal R}\) is the set of relationships). \(t_{e}\) and \(t_{\tilde{e}}\) stand for the hierarchical types of left entity \(e\) and right entity \(\tilde{e}\) respectively. (2) Since the relationship \(r\) remains unchanged in replacement, we build two differentiated embeddings for the \(i\)-th relationship \(r_{i}\) in two embedding spaces: \({\bf r}_{i}^{*}(\in\mathbb{R}^{\kappa})\) in entity space and \({\bf r}_{i}^{o}~{}(\in\mathbb{R}^{\ell})\) in entity type space. (3) Given entity type triple \((t_{e},r,t_{\tilde{e}})\), under translation assumption 2 as in Bordes et al. (2013), we have: \({\bf t}_{\tilde{e}}-{\bf r}^{o}\simeq{\bf t}_{e}\). Hence, the scoring function is defined as:

Footnote 2: We chose TransE in this paper, and it is not difficult for other enhanced translation-based methods to model triple knowledge, such as Trans(H, R, D and G) Wang et al. (2017).

\[{\bf S}_{trt}(t_{e},r,t_{\tilde{e}})=\|{\bf t}_{e}+{\bf r}^{o}-{\bf t}_{\tilde {e}}\|_{\ell 2}^{2}~{}~{}, \tag{2}\]

where \({\bf t}_{e},{\bf r}^{o},{\bf t}_{\tilde{e}}\in\mathbb{R}^{\ell}\). The model returns a lower score if the two entity types is close under this relationship and a higher one otherwise.

Fig.2 shows an illustration of E2T and TRT.

### Implementation for Entity Type Prediction

Our framework can be used for entity type prediction in the following way. First, for each entity

Figure 2: Simple illustration of E2T and TRT.

 

that appears in the testing set, a prediction by E2T is performed with:

\[\hat{t}_{e}=\operatorname*{arg\,min}_{t\in\mathcal{T}}~{}~{}\mathbf{S}_{e2t}(e ,t). \tag{3}\]

In addition, a composite score (E2T+TRT) by connecting entity type instances and entity type triples with embedding model, which we call ConnectE 3, is defined as follows:

Footnote 3: We also call it Connect-(E2T+TRT), and use ConnectE-(E2T+0) to denote E2T for uniformity in the experiments.

\[\begin{split}&\mathbf{S}_{e2t+trt}(e,t_{e})=\lambda\cdot\mathbf{ S}_{e2t}(e,t_{e})+\\ &(1-\lambda)\cdot\Big{\{}\frac{1}{|P|}\sum_{t_{\ell}\in P} \mathbf{S}_{trt}(t_{e},r,t_{\bar{e}})\\ &\qquad\quad\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \; 

according to \({\cal H}\). The generated datasets are called FB15kTRT(full) and YAG043kTRT(full). Second, considering about the scalability of the proposed approach for full KGs, we further modify the generation method of type triples, which is the major training bottleneck. We discard newly generated ones with low-frequency (i.e. #frequency = 1). After that the size of both FB15kTRT(full) and YAGO43kTRT(full) decreased by about 90%, called FB15kTRT(disc.) and YAGO43kTRT(disc.) respectively. The statistics of the datasets are showed in Table 2. For saving space, we put more data processing details (include cleaning \({\cal H}\), building \({\cal Z}\), etc.) on our github website.

### Entity Type Prediction

This task concentrates to complete a pair (_entity, entity type_) when its type is missing, which aims to verify the capability of our model for inferring missing entity type instances.

**Evaluation Protocol.** We focus on entity type prediction determined by Formula (3) and (4). We use ranking criteria for evaluation. Firstly for each test pair, we remove the type and replace it by each of the types in \({\cal T}\) in turn. The function value of the negative pairs would be computed by the related models and then sorted by ascending order. We can obtain the exact rank of the correct type in the candidates. Finally, we use two metrics for comparison: (1) the mean reciprocal rank (MRR), and (2) the proportion of correct entities ranked in the top 1/3/10 (HITS@1/3/10)(%). Since the evaluation setting of "Raw" is not as accurate as "Filter" [7], we only report the experimental results with latter setting in this paper.

\[MRR=\frac{1}{|C|}\sum^{|C|}_{i=1}\frac{1}{rank_{i}}~{},\]

where \(C\) is a set of test pairs, and \(rank_{i}\) is the rank position of the true entity type for the \(i\)-th pair.

Implementation.The results of entity type prediction are shown in Table 3, where the results for the baselines are directly taken from original literature [8]. We do not choose LM and PEM [6] as baselines since they do not utilize triple knowledge, thus it is not fair to compare with them. For training our model, we select the learning rate \(\alpha\in\{\)0.1, 0.05, 0.001\(\}\), the margins \(\gamma_{1},\gamma_{2},\gamma_{3}\in\{0.5,1,2,5,10\}\), the embedding dimension pairs \((\kappa,\ell)\in\{\)(100, 50), (150, 75), (200, 100), (250, 125)\(\}\), and the weight \(\lambda\in\{\)0.5, 0.65, 0.85, 0.95\(\}\). We use negative sampling, and gradient descent with AdaGrad as our optimization approach to improve convergence performance. During the initialization process, each embedding vector of the entities, entity types and relationships is initialized with a random number following a uniform distribution \(-\sqrt{6}/(m+n)\), where \(n\in\{\)#Ent, #Type, #Rel\(\}\) and \(m\in\{\kappa,\ell\}\). During the whole training process, we normalize the entity embeddings after each epoch.

We select the parameters based on MRR in valid dataset. The optimal configurations are: \(\{\alpha=0.1,\gamma_{1}=\gamma_{2}=\gamma_{3}=2,\kappa=200,\ell=100,\lambda=0.85\}\) on FB15k/ET/TRT; \(\{\alpha=0.1,\gamma_{1}=\gamma_{2}=\gamma_{3}=1,\kappa=250,\ell=125,\lambda= 0.85\}\) on YAGO43k/ET/TRT. We run 800 epochs on both datasets, and the batch size is 4096.

Experimental Results.We can see from Table 3 that our ConnectEs outperform all baselines for entity type prediction in terms of all metrics on FB15kET and YAGO43kET. It confirms the capability of ConnectEs in modeling with local typing and global triple knowledge and inferring missing entity type instances in KGs. The model ConnectE(E2T+TRT)(full) achieves the highest scores.

Analysis.(1) In E2T, we utilize a mapping matrix M which compresses entity embeddings into type embedding space, considering that entity type embedding represents common information of all the entities which belong to this type. The type embedding should be in a sharing subspace of entity embeddings. The experimental results of E2T compared with the baselines demonstrate that this assumption would be quite reasonable. (2) In E2T+TRT, we build new type-relation-type data, and then connect them with entity type instances. This approach provides more direct useful information to (weakly) supervise entity type prediction. For example, given a fact that head entity _Barack Obama_ belongs to type _/people/person 

and the relationship _born_in_, we could make the best guess of the type of tail entity _Honolulu_ as _location/location_. Hence, the addition of type triples in ConnectE-(E2T+TRT) provides superior performance than ConnectE-(E2T+0). (3) Concerning about the scalability of our approach for big KGs, we utilize FB15kTRT(disc.) and YAGO43KTRT(disc.) for prediction, the training time of which reduced by 90% as the training data size decreased by 90%. Moreover, the results of ConnectE-(E2T+TRT)(disc.) show that it's comparable with the best ConnectE-(E2T+TRT)(full).

### Entity Type Classification

This task aims to judge whether each entity type instance in testing data holds or not, which could be viewed as a binary classification problem.

**Evaluation Protocol.** Since there are no explicit negative entity type instances in existing KGs, in order to create datasets for classification, we build negative facts by randomly switching type from entity type pairs in validation and testing set with equal number of positive and negative examples. Inspired by the evaluation metric of triple classification in Socher et al. (2013), we calculate the scores of all entity type instances based on model energy function, and rank all instances in testing set with these scores. Those instances with lower scores are considered to be true. We use **precision/recall curves** to show the performances of all models. Moreover, we also compare the **accuracy** among different models. We first use validate set to find best threshold \(\eta\). For instance, if the model score \({\bf S}_{e2t+trt}(e,t_{e})\leq\eta\) in classification, the entity type instance will be classified to be positive, otherwise to be negative. The final accuracy is based on how many facts are classified correctly.

**Implementation.** We utilize the source codes and parameter settings of several baselines provided by Moon et al. (2017) for this task. The optimal parameter settings for our proposed models are: \(\{\alpha=0.1,\gamma_{1}=\gamma_{2}=\gamma_{3}=2,\kappa=200,\ell=100,\lambda=0 8.85\}\) on FB15kT; \(\{\alpha=0.1,\gamma_{1}=\gamma_{2}=\gamma_{3}=1,\kappa=250,\ell=125,\lambda= 0.85\}\) on YAGO43kET. In both datasets, we learn all the training data for 800 epochs and the batch size is 4096. After training, we firstly draw PR-curves with dynamic thresholds. We select the best threshold based on the accuracy in valid dataset, which is used to calculate the accuracy in test dataset.

**Experimental Results.** We draw the PR-curves for type classification task on both datasets in Fig.3. Note that we only report the results of ConnectE-(E2T+TRT)(disc.) not ConnectE-(E2T+TRT)(full), since the learning speed of the former is much more faster than the latter and its results are close to the best results of the latter. We can see from Fig.3 that when the recall rate is between 0.88 \(\sim\) 0.97, ConnectE-(E2T+TRT)(disc.) model could achieve the highest precision rate on FB15kT. In other ranges, our ConnectE-(E2T+TRT)(disc.) model also shows comparable performance. The result is consistent on YAGO43kET. Specifically, ConnectE-(E2T+TRT)(disc.) achieves the best F1 score of 94.66% when recall = 94.27% and precision = 95.05% on FB15kT. Also, ConnectE-(E2T+TRT)(disc.) surpasses other models and gets F1 score of 92.13% when precision = 93.18% and recall = 91.11% on YAGO43kET. It confirms the capability of our model, for they could not only infer missing types in KGs, but also perform well in KG entity type classification.

Table 4 demonstrates the evaluation accuracy results of entity type classification, from which we can observe that: (1) On FB15kT, ConnectE-(E2T+TRT)(disc.) achieves the best accuracy score (94.49%). Compared to the mostly related model ETE, our model shows 0.48% absolute performance improvement. On YAGO43kET, ConnectE-(E2T+TRT)(disc.) model outperforms other models as well. The improvement of our model com



pared to ETE is almost 1.51%. (2) Comparing to the improvement on YAGO43kET, the advantage ConnectE-(E2T+TRT)(disc.) has over ConnecticutE-(E2T+0) in this task on FB15kET seems to be insignificant, which indicates that the type triples in FB15kTRT have fewer contribution on entity type classification than ones in YAGO43kTRT. It may be partially caused by the fact that the number of relations in YAGO43k (#Rel=37) is far less than that in FB15k (#Rel=1,345), which could considerably influence the effectiveness of the type-relation-type training set. Due to the rareness of relationships in YAGO43k, each entity usually connects with a large number of other entities through one single relationships, which means that the magnitude of \(|P|\) and \(|Q|\) in the composite model scoring function are large. After averaging in ConnectE-(E2T+TRT)(disc.), it could achieve more stable and significant results on YAGO43kET.

### Case Study

Table 5 shows the examples of entity type prediction by our model from _FB15k/ET/TRT_, which demonstrate our motivation of Mech. 2 that head type and tail type really maintain the relationship between head entity and tail entity. Given entity _Peter Berg_, TRT can find HITS@1 type prediction _/people/person_ for it via the existing entity type assertion (_New Youk, /location/location_) and the relationship (_loc.loc./loc./people_born_here_) between them, i.e. _PeterBerg - NewYork + location/location= /people/person_.

## 5 Conclusion and Future Work

In this paper, we described a framework for leveraging global triple knowledge to improve KG entity typing by training not only on (_entity, entity type_) assertions but also using newly generated (_head type, relationship, tail type_) type triples. Specifically, we propose two novel embedding-based models to encode entity type instances and entity type triples respectively. The connection of both models is utilized to infer missing entity type instances. The empirical experiments demonstrate the effectiveness of our proposed model. Our modeling method is general and should apply to other type-oriented tasks. Next, we are considering to use this framework to conduct KG entity type noise detection.

Figure 3: **Entity type classification results (Precision/Recall Curve). Evaluate on FB15kET, YAGO43kET.**





## Acknowledgments

The authors would like to thank all anonymous reviewers for their insightful comments. We also want to thank Zhiyuan Liu (Tsinghua University) and Linmei Hu (BUPT) for their useful suggestions and comments on early drafts. This work was supported by the National Natural Science Foundation of China under Grant No.61922085, 61906159, the Sichuan Science and Technology Program under Grant No.2018JY0607, the Fundamental Research Funds for the Central Universities under Grant No.JJK2003008, Fintech Innovation Center, and Financial Intelligence and Financial Engineering Key Laboratory of Sichuan Province.

## References

* Abhishek et al. (2017) Abhishek Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. In _Proceedings of EACL_, page 797807.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Grostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of EMNLP_, pages 1533-1544.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _Proceedings of SIGMOD_, pages 1247-1250.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Proceedings of NeurIPS_, pages 2787-2795.
* Choi et al. (2018) Eunsol Choi, Omer Levy, Yejin Choi, and Luke S. Zettlemoyer. 2018. Ultra-fine entity typing. In _Proceddings of ACL_.
* Dettmers et al. (2017) Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. 2017. Convolutional 2d knowledge graph embeddings. In _Proceedings of AAAI_, pages 1811-1818.
* Ding et al. (2018) Boyang Ding, Quan Wang, Bin Wang, and Li Guo. 2018. Improving knowledge graph embedding using simple constraints. In _Proceedings of ACL_, pages 110-121, Melbourne, Australia.
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In _Proceedings of SIGKDD_, pages 601-610.
* Elsahar et al. (2018) Hady Elsahar, Christophe Gravier, and Frederique Laforest. 2018. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. In _Proceedings of NAACL_, pages 218-228.
* Gupta et al. (2017) Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In _Proceedings of EMNLP_, pages 2681-2690.
* Hajishirizi et al. (2013) Hannaneh Hajishirizi, Leila Zilles, Daniel S. Weld, and Luke Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In _Proccddings of EMNLP_, pages 289-299.
* Hao et al. (2019) Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, and Wei Wang. 2019. Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts. In _Proceedings of KDD 2019_.
* Jain et al. (2018) Prachi Jain, Pankaj Kumar, and Soumen Chakrabarti. 2018. Type-sensitive knowledge base inference without explicit type supervision. In _Proceedings of ACL_.
* Ji et al. (2016) Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2016. Knowledge graph completion with adaptive sparse transfer matrix. In _Procddings of AAAI_.
* Jin et al. (2018) Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2018. Attributed and predictive entity embedding for fine-grained entity typing in knowledge bases. In _Proceedings of COLING 2018_, pages 282-292.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _arXiv preprint arXiv:1412.6980_.
* Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In _Proceedings of AAAI_, pages 2181-2187.
* Lv et al. (2018) Xin Lv, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Differentiating concepts and instances for knowledge graph embedding. In _Proceedings of EMNLP 2018_, page 19711979.
* Moon et al. (2017) Chang Moon, Paul Jones, and Nagiza F. Samatova. 2017. Learning entity type embeddings for knowledge graph completion. In _Proceedings of CIKM_, pages 2215-2218.
* Nathani et al. (2019) Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In _Proceedings of ACL_.
* Neelakantan et al. (2015) Arvind Neelakantan, Ming-Wei Chang, and . 2015. Inferring missing entity type instances for knowledge base completion: New dataset and methods. In _Proceedings of NAACL 2012_, page 515525

Maximilian Nickel, Lorenzo Rosasco, and Tomasao Poggio. 2016. Holographic embeddings of knowledge graphs. In _Proceedings of AAAI_, pages 1955-1961.
* Nickel et al. (2011) Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In _Proceedings of ICML_, pages 809-816.
* Pantel et al. (2012) Patrick Pantel, Thomas Lin, and Michael Gamon. 2012. Mining entity types from query logs via user intent modeling. In _Proceedings of ACL_, pages 563-571.
* Socher et al. (2013) Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In _Proceedings of NeurIPS_, pages 926-934.
* Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In _Proceedings of WWW_, pages 697-706.
* Wang et al. (2017) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. _TKDE_, 29(12):2724-2743.
* Wang et al. (2014) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In _Proceedings of AAAI_, pages 1112-1119.
* Xie et al. (2016) Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016. Representation learning of knowledge graphs with hierarchical types. In _Proceedings of IJCAI_, pages 2965-2971.
* Xu and Barbosa (2018) Peng Xu and Denilson Barbosa. 2018. Neural fine grained entity type classification with hierarchy aware loss. In _Proceedings of NAACL_.
* Yaghoobzadeh et al. (2018) Yadollah Yaghoobzadeh, Heike Adel, and Hinrich Schutze. 2018. Corpus-level fine-grained entity typing. _Journal of Artificial Intelligence Research_, 61:835-862.
* Yao et al. (2013) Limin Yao, Sebastian Riedel, , and Andrew McCallum. 2013. Universal schema for entity type prediction. In _Proceedings of the 2013 Workshop on Automated Knowledge Base Construction_, pages 79-84.
* Yogatama et al. (2015) Dani Yogatama, Daniel Gillick, and Nevena Lazic. 2015. Embedding methods for fine grained entity type classification. In _Proceedings of ACL_, page 291296.
* Yuan and Downey (2018) Zheng Yuan and Doug Downey. 2018. Otyper: A neural architecture for open named entity typing. In _Proceddings of AAAI_.
* Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In _Proceedings of COLING_, pages 23-29.
* Zhang et al. (2018) Richong Zhang, Fanshuang Kong, Chenyue Wang, and Yongyi Mao. 2018. Embedding of hierarchically typed knowledge bases. In _Proccddings of AAAI_.
* Zhou et al. (2018) Ben Zhou, Daniel Khashabi, Chen-Tse Tsai, and Dan Roth. 2018. Zero-shot open entity typing as typecompatible grounding. In _Procedddings of EMNLP_, pages 2065-2076.