SlowGuess/ACL_2020_OCR huggingface dataset:
Yu Zhao $^{1,*}$ , Anxiang Zhang $^{2,*}$ , Ruobing Xie $^{3}$ , Kang Liu $^{4,5}$ , Xiaojie Wang $^{6}$
$^{1}$ Fintech Innovation Center, School of Economic Information Engineering, Southwestern University of Finance and Economics, Chengdu, China
$^{2}$ School of Computer Science, Carnegie Mellon University, Pittsburgh, USA
<sup>3</sup>WeChat Search Application Department, Tencent, Beijing, China
$^{4}$ National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
$^{5}$ University of Chinese Academy of Sciences, Beijing, 100049, China
$^{6}$  School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China
Knowledge graph (KG) entity typing aims at inferring possible missing entity type instances in KG, which is a very significant but still under-explored subtask of knowledge graph completion. In this paper, we propose a novel approach for KG entity typing which is trained by jointly utilizing local typing knowledge from existing entity type assertions and global triple knowledge from KGs. Specifically, we present two distinct knowledge-driven effective mechanisms of entity type inference. Accordingly, we build two novel embedding models to realize the mechanisms. Afterward, a joint model with them is used to infer missing entity type instances, which favors inferences that agree with both entity type instances and triple knowledge in KGs. Experimental results on two real-world datasets (Freebase and YAGO) demonstrate the effectiveness of our proposed mechanisms and models for improving KG entity typing. The source code and data of this paper can be obtained from: https://github.com/Adam1679/ConnectE
The past decade has witnessed great thrive in building web-scale knowledge graphs (KGs), such as Freebase (Bollacker et al., 2008), YAGO (Suchanek et al., 2007), Google Knowledge Graph (Dong et al., 2014), which usually consists of a huge amount of triples in the form of (head entity, relation, tail entity) (denoted  $(e,r,\tilde{e})$ ). KGs usually suffer from incompleteness and miss important facts, jeopardizing their usefulness in downstream tasks such as question answering (Elsahar et al., 2018), semantic parsing (Berant et al., 2013), relation classification (Zeng et al., 2014). Hence, the task of
Figure 1: Effective mechanisms of entity type inference with local typing knowledge and global triple knowledge.
knowledge graph completion (KGC, i.e. completing knowledge graph entries) is extremely significant and attracts wide attention.
This paper concentrates on KG entity typing, i.e. inferring missing entity type instances in KGs, which is an important sub-problem of KGC. Entity type instances, each of which is in the formed of (entity, entity type) (denoted  $(e,t)$ ), are essential entries of KGs and widely used in many NLP tasks such as relation extraction (Zhang et al., 2018; Jain et al., 2018), coreference resolution (Hajishirzi et al., 2013), entity linking (Gupta et al., 2017). Most previous works of KGC focus on inferring missing entities and relationships (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015; Dettmers et al., 2017; Ding et al., 2018; Nathani et al., 2019), paying less attention to entity type prediction. However, KGs also usually suffer from entity types incompleteness. For instance,  $10\%$  of entities in  $FB15k$  (Bordes et al., 2013), which have the /music/artist type, miss the /people/person type (Moon et al., 2017). KG entity type incompleteness leads to some type-involved algorithms in KG-driven tasks grossly inefficient or even unavailable.
To solve KG entity type incompleteness issue, in this paper we propose a novel embedding methodology to infer missing entity type instances that employs not only local typing knowledge from entity type assertions, as most conventional mod
* Equal Contribution. Corresponding author: Y. Zhao (zhaoyu@swufe.edu.cn).
6419
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6419-6428 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
els do, but also leverages global triple knowledge from KGs. Accordingly, we build two distinct knowledge-driven type inference mechanisms with these two kinds of structural knowledge.
Mechanism 1. Missing entity types of an entity can be found from other entities that are close to the entity in the embedding space, using local typing knowledge as in Fig.1(Mech.1).
Mechanism 2. Missing entity types of an (head or tail) entity can be inferred from the types of other (tail or head) entities through their relationships, using global triple knowledge as in Fig.1(Mech.2).
The main idea behind Mech.1 is based on the observation that the learned entities' embeddings by conventional KG embedding methods (Ji et al., 2016; Xie et al., 2016) cluster well according to their types in vector space. For instance, in Fig.1(Mech.1), given an entity Barack Obama, it's missing hierarchical type /people/person can be induced by the given hierarchical type of similar entity Donald Trump. In addition, the key motivation behind Mech.2 is that the relationship shall remain unchanged if the entities in a triple fact are replaced with their corresponding hierarchical types. For instance, given a global triple fact (Barack Obama, born_in, Honolulu), under this assumption, we can induce a new type triple (/people/person, born_in, location/location)^1. Formally, Honolulu - Barack Obama = location/location - people/person (= born_in), which can be used to infer missing entity types, e.g. (Barack Obama, type=? ) via Barack Obama - Honolulu + location/location = /people/person, as Mech.2 does. Fig.1 demonstrates a simple illustration of effective mechanisms of entity type inference. Both mechanisms are utilized to build our final composite model.
Specifically, we build two embedding models to realize the two mechanisms respectively. First, considering entities and entity types are completely distinct objects, we build two distinct embedding spaces for them, i.e., entity space and entity type space. Accordingly, we encode  $(e,t)$  entity type instance by projecting the entity from entity space to entity type space with mapping matrix  $\mathbf{M}$ , hence we have (1):  $\boxed{\mathbf{M} \cdot \mathbf{e} \simeq \mathbf{t}}$ , called E2T. Moreover, we learn the plausibility of  $(t_e,r,t_{\tilde{e}})$  global type triple by newly generalizing from  $(e,r,\tilde{e})$  global
triple fact, even though this type triple is not present originally. Following translating assumption (Bordes et al., 2013), we have (2):  $\boxed{\mathbf{t}_{\tilde{e}} - \mathbf{r}^{\circ}\simeq \mathbf{t}_{e}}$  called TRT. E2T and TRT are the implementation models of the two mechanisms. Fig.2 demonstrates a brief illustration of our models. A ranking-based embedding framework is used to train our models. Thereby, entities, entity hierarchical types, and relationships are all embedded into low-dimensional vector spaces, where the composite energy score of both E2T and TRT are computed and utilized to determine the optimal types for (entity, entity type=? ) incomplete assertions. The experimental results on real-world datasets show that our composite model achieves significant and consistent improvement compared to all baselines in entity type prediction and achieves comparable performance in entity type classification.
- We propose a novel framework for inferring missing entity type instances in KGs by connecting entity type instances and global triple information and correspondingly present two effective mechanisms.
- Under these mechanisms, we propose two novel embedding-based models: one for predicting entity types given entities and another one to encode the interactions among entity types and relationships from KGs. A combination of both models are utilized to conduct entity type inference.
- We conduct empirical experiments on two real-world datasets for entity type inference, which demonstrate our model can successfully take into account global triple information to improve KG entity typing.
Entity typing is valuable for many NLP tasks (Yaghoobzadeh et al., 2018), such as knowledge base population (Zhou et al., 2018), question answering (Elsahar et al., 2018), etc. In recent years, researchers attempt to mine fine-grained entity types (Yogatama et al., 2015; Choi et al., 2018; Xu and Barbosa, 2018; Yuan and Downey, 2018) with external text information, such as web search query logs (Pantel et al., 2012), the textual surface patterns (Yao et al., 2013), context representation (Abhishek et al., 2017), Wikipedia (Zhou et al.,
For more clarity, we represent it as (/location/location, born_in $^{-1}$ , /people/person) in Fig.1(Mech.2).
6420
Table 1: Entity type embedding models.
Models	Energy function	Parameters	Sources	Training strategy
Se2t(e,t)	Striple(·)
LM (Neelakantan et al., 2015)	eTt	N/A	e, t ∈ Rκ	entity type instances	N/A
PEM (Neelakantan et al., 2015)	eTVTt	N/A	e ∈ Rκ, t ∈ Rℓ, U ∈ Rκ×d, V ∈ Rℓ×d	entity type instance	N/A
RESCAL (Nickel et al., 2011)	N/A	eTMr	e, e ∈ Rκ, Mr ∈ Rκ×κ	mixed triple knowledge	syn.
RESCAL-ET (Moon et al., 2017)	||e - t||1	eTMr	e, e, t ∈ Rκ, Mr ∈ Rκ×κ	entity type inst./ triple know.	asyn.
HOLE (Nickel et al., 2016)	N/A	rT(e★e)	e, r, e ∈ Rκ	mixed triple knowledge	syn.
HOLE-ET (Moon et al., 2017)	||e - t||1	rT(e★e)	e, r, e, t ∈ Rκ	entity type inst./ triple know.	asyn.
TransE (Bordes et al., 2013)	N/A	||e + r - e||	e, r, e ∈ Rκ	mixed triple knowledge	syn.
TransE-ET (Moon et al., 2017)	||e - t||1	||e + r - e||	e, r, e, t ∈ Rκ	entity type inst./ triple know.	asyn.
ETE (Moon et al., 2017)	||e - t||1	||e + e + c - r||	e, r, e, c, t ∈ Rκ	entity type inst./ triple know.	asyn.
ConnectE (our proposed)	||M·e - t||2	||e + r★ - e||2, ||te + r° - te||2	e, r* ∈ Rκ, t, r° ∈ Rℓ, M ∈ Rℓ×κ	entity type inst./ triple know.	syn.
2018). Despite their success, existing methods rely on additional external sources, which might not be feasible for some KGs.
To be more universal, Neelakantan et al. (2015) propose two embedding models, i.e. linear model (LM) and projection embedding model (PEM), which can infer missing entity types only with KG itself. Although PEM has more expressive power than LM, however, both of them ignore global triple knowledge, which could also be helpful for encoding entity type assertions via shared entities' embeddings. To address this issue, Moon et al. (2017) propose a state-of-the-art model (ETE) to combine triple knowledge and entity type instances for entity type prediction, and build two entity type embedding methodologies: (1) Synchronous training: treat (entity, entity type) assertions as special triple facts that have a unique relationship "rdfs:type", e.g. (Barack Obama, "rdfs:type", person), and encode all mixed triple facts (original triple data fused with all generated special ones) by conventional entity relation embedding models, such as RESCAL (Nickel et al., 2011), HOLE (Nickel et al., 2016) and TransE (Bordes et al., 2013). (2) Asynchronous training: first learn the entities' embeddings  $\mathbf{e}$  by conventional entity relation embedding models mentioned above, and then only update entity types' embeddings  $\mathbf{t}$  for  $\min \| \mathbf{e} - \mathbf{t}\|_{\ell 1}$  while keeping  $\mathbf{e}$  fixed, called RESCAL-ET, HOLE-ET, TransE-ET and ETE. Although these approaches expect to explore global triple knowledge for entity type prediction, they still lack of expressive ability due to its simplicity of embeddings. In addition, they irrationally assume both the embeddings of entities and entity types being in the same latent space
$(\in \mathbb{R}^{\kappa})$ . Since entities and entity types are completely distinct objects, it may not be reasonable to represent them in a common semantic space.
In this paper, we introduce an enhanced KG entity type embedding model with better expressing and reasoning capability considering both local entity typing information and global triple knowledge in KGs. Note that incorporating more external information (Jin et al., 2018; Neelakantan et al., 2015) is not the main focus in this paper, as we only consider the internal structural information in KGs instead, which correspondingly makes our work much more challenging but also more universal and flexible due to the limited information. Recently, (Lv et al., 2018; Hao et al., 2019) also attempt to embedding structural information in KG. However, the goals and models are very different from ours. They encodes the concepts, not hierarchical types. On the contrary, we focus on the latter not the former. Table 1 summarizes the energy functions and other different settings of entity type embedding models.
3 Embedding-based Framework
We consider a KG containing entity type instances of the form  $(e,t)\in \mathcal{H}$  ( $\mathcal{H}$  is the training set consists of lots of (entity, entity type) assertions), where  $e\in \mathcal{E}$  ( $\mathcal{E}$  is the set of all entities) is an entity in the KG with the type  $t\in \mathcal{T}$  ( $\mathcal{T}$  is the set of all types). For example,  $e$  could be Barack Obama and  $t$  could be /people/person. As a single entity can have multiple types, entities in KG often miss some of their types. The aim of this work is to infer missing entity type instances in KGs.
Our work concerns energy-based methods,
6421
Figure 2: Simple illustration of E2T and TRT.
which learn low-dimensional vector representations (embeddings) of atomic symbols (i.e. entities, entity hierarchical types, relationships). In this framework, we learn two submodels: (1) one for predicting entity types given entities, and (2) another one to encode the interactions among entity types and relationships from KGs. The joint action of both models in prediction allows us to use the connection between triple knowledge and entity type instances to perform KG entity typing.
3.1 E2T: Mapping Entities to Types
The first model (E2T) of the framework concerns the learning of a function  $\mathbf{S}_{e2t}(e,t)$  with local typing knowledge from entity type instances, which is designed to score the similarity of an entity  $e$  and a type  $t$ . The main ideas behind this model are as follows: (1) Since the learned entity embeddings cluster well when they have the same or similar types, therefore, it is rather intuitive that the entity type embedding represents the projective common concept representation of a cluster of entities, i.e.,  $f_{proj}(\mathbf{e}) \simeq \mathbf{t}_e, \forall e \in \mathcal{E}$ .  $\mathbf{e} (\in \mathbb{R}^{\kappa})$  is the embedding of the entity  $e$ ,  $\mathbf{t}_e (\in \mathbb{R}^\ell)$  is the embedding of the type  $t_e$ . The entity type embedding represents common information of their entities, it thus should have fewer variates, i.e.,  $\ell < \kappa$ . (2) Since the entities and entity types are totally distinct objects, we respectively build two embedding space for them, i.e., entity space and entity type space. (3) Inspired by the previous work TranSparse (Ji et al., 2016) projecting entities from entity space to relation space with operation matrix  $\mathbf{M}$ , which we adapted, replacing relation space with entity type space, we thus define  $f_{proj}(\mathbf{e}) = \mathbf{M} \cdot \mathbf{e} (\simeq \mathbf{t}_e)$ . Therefore, this model consists of first projecting entity embedding into entity type space, and then computing a similarity measure between this projection and an entity type embedding. The scoring
function of E2T given  $(e,t)$  is:
$$
\mathbf {S} _ {e 2 t} (e, t) = \left\| \mathbf {M} \cdot \mathbf {e} - \mathbf {t} \right\| _ {\ell 2} ^ {2}, \tag {1}
$$
where  $\mathbf{M} \in \mathbb{R}^{\ell \times \kappa}$  is a transfer matrix mapping entity embeddings into entity type space. The score is expected to be lower for a golden entity type instance and higher for an incorrect one.
3.2 TRT: Encoding Triples in KGs
Using only entity type instances for training ignores much of relational knowledge that can leverage from triple facts in KGs. In order to connect this relational data with our model, we propose to learn entity type and relationship embeddings from global triple knowledge from KGs. The key motivations behind this model are: (1) As mentioned above, the entities cluster well according to their types. Therefore, we believe that an essential premise of a triple (head entity, relationship, tail entity) holds is that its corresponding entity types should first conform to this relationship. Hence, we can build a new entity type triple (head type, relationship, tail type) by replacing both head entity and tail entity with their corresponding types: i.e.  $(e,r,\tilde{e})\stackrel {replace}{\longrightarrow}(t_e,r,t_{\tilde{e}})$ .  $(e,r,\tilde{e})\in \mathcal{D}$ ,  $\mathcal{D}$  is the training set consists of a lot of triples.  $r\in \mathcal{R}$  ( $\mathcal{R}$  is the set of relationships).  $t_e$  and  $t_{\tilde{e}}$  stand for the hierarchical types of left entity  $e$  and right entity  $\tilde{e}$  respectively. (2) Since the relationship  $r$  remains unchanged in replacement, we build two differentiated embeddings for the  $i$ -th relationship  $r_i$  in two embedding spaces:  $\mathbf{r}_i^\star (\in \mathbb{R}^\kappa)$  in entity space and  $\mathbf{r}_i^\circ (\in \mathbb{R}^\ell)$  in entity type space. (3) Given entity type triple  $(t_e,r,t_{\tilde{e}})$ , under translation assumption as in (Bordes et al., 2013), we have:  $\mathbf{t}_{\tilde{e}} - \mathbf{r}^{\circ}\simeq \mathbf{t}_{e}$ . Hence, the scoring function is defined as:
$$
\mathbf {S} _ {t r t} (t _ {e}, r, t _ {\tilde {e}}) = \left\| \mathbf {t} _ {e} + \mathbf {r} ^ {\circ} - \mathbf {t} _ {\tilde {e}} \right\| _ {\ell 2} ^ {2}, \tag {2}
$$
where  $\mathbf{t}_e, \mathbf{r}^\circ, \mathbf{t}_{\tilde{e}} \in \mathbb{R}^\ell$ . The model returns a lower score if the two entity types is close under this relationship and a higher one otherwise.
Fig.2 shows an illustration of E2T and TRT.
3.3 Implementation for Entity Type Prediction
Our framework can be used for entity type prediction in the following way. First, for each entity  $e$
2We chose TransE in this paper, and it is not difficult for other enhanced translation-based methods to model triple knowledge, such as Trans(H, R, D and G) (Wang et al., 2017).
6422
that appears in the testing set, a prediction by E2T is performed with:
$$
\hat {t} _ {e} = \underset {t \in \mathcal {T}} {\arg \min } \mathbf {S} _ {e 2 t} (e, t). \tag {3}
$$
In addition, a composite score (E2T+TRT) by connecting entity type instances and entity type triples with embedding model, which we call ConnectE<sup>3</sup>, is defined as follows:
$$
\begin{array}{l} \mathbf {S} _ {e 2 t + t r t} (e, t _ {e}) = \lambda \cdot \mathbf {S} _ {e 2 t} (e, t _ {e}) + \\ (1 - \lambda) \cdot \left\{\frac {1}{| P |} \sum_ {t _ {\tilde {e}} \in P} \mathbf {S} _ {t r t} (t _ {e}, r, t _ {\tilde {e}}) \right. \\ \left. + \frac {1}{| Q |} \sum_ {t _ {\bar {e}} \in Q} \mathbf {S} _ {t r t} (t _ {\bar {e}}, r, t _ {e}) \right\}, \\ \end{array}
$$
where  $\lambda$  is a hyperparameter for the trade-off.  $P = \{t_{\tilde{e}}|t_{\tilde{e}}\in \mathcal{T},(e,r,\tilde{e})\in \mathcal{D}\}$  (i.e. given  $e$  is head entity,  $P$  is the set of all corresponding tail entities' types.), and  $Q = \{t_{\bar{e}}|t_{\bar{e}}\in \mathcal{T},(\bar{e},r,e)\in \mathcal{D}\}$  (i.e. given  $e$  is tail entity,  $Q$  is the set of all corresponding head entities' types.).  $|P|$  and  $|Q|$  represent the total number of entity types in  $P$  and  $Q$  respectively. A prediction is performed with:
$$
\hat {t} _ {e} = \underset {t _ {e} \in \mathcal {T}} {\arg \min } \mathbf {S} _ {e 2 t + t r t} (e, t _ {e}). \tag {4}
$$
Hence, our final composite model ConnectE(E2T+TRT) favors predictions that agree with both entity type instances and global triple information in KGs.
We use ranking loss algorithm for training ConnectE-(E2T+TRT), in which the parameter set  $\Theta = \{\mathbf{E},\mathbf{T},\mathbf{R}^{\star},\mathbf{R}^{\circ},\mathbf{M}\}$ .  $\mathbf{E},\mathbf{T}$  stand for the collection of all entities' and types' embeddings respectively.  $(\mathbf{R}^{\star},\mathbf{R}^{\circ})$  denotes the collections of relationships' differentiated embeddings. The ranking objectives are designed to assign lower scores to true facts (including  $(e,r,\tilde{e})$  triple facts,  $(e,t)$  entity type instances and  $(t_e,r,t_{\tilde{e}})$  type triples) versus any corrupt ones. We build three sub-objective functions, i.e.,  $\mathbf{J}_1,\mathbf{J}_2,\mathbf{J}_3$ , and implement dynamic optimization strategy, i.e., fix a partial of parameters and only update the rest when minimizing each function. (1)  $\mathbf{J}_1$ : We choose TransE (see Bordes et al. (2013)) to model triple facts as  $\mathbf{S}(e,r,\tilde{e})$ , in which we update the embeddings of entities  $(\forall \mathbf{e}\in \mathbf{E})$  and the embeddings of relationships
$(\forall \mathbf{r}^{\star}\in \mathbf{R}^{\star})$  .2)  $\mathbf{J}_2$  We only update the embeddings of entity types  $(\forall \mathbf{t}\in \mathbf{T})$  and projecting matrix M, not the entities' embeddings that have been trained in  $\mathbf{J}_1$  3)  $\mathbf{J}_3$  :We only update the embeddings of relationships  $(\forall \mathbf{r}^{\circ}\in \mathbf{R}^{\circ})$  while keeping the entity types' embeddings fixed. The training is performed using Adagrad (Kingma and Ba, 2014). All embeddings in  $\Theta$  are initialized with uniform distribution. The procedure, from  $\mathbf{J}_1,\mathbf{J}_2$  to  $\mathbf{J}_3$  , is iterated for a given number of iterations. We have:
$$
\mathbf {J} _ {1} = \sum_ {\mathcal {D}} \sum_ {\mathcal {D} ^ {\prime}} [ \gamma_ {1} + \mathbf {S} (e, r, \tilde {e}) - \mathbf {S} (e ^ {\prime}, r, \tilde {e} ^ {\prime}) ] _ {+},
$$
$$
\mathbf {J} _ {2} = \sum_ {\mathcal {H}} \sum_ {\mathcal {H} ^ {\prime}} \left[ \gamma_ {2} + \mathbf {S} _ {e 2 t} \left(e, t _ {e}\right) - \mathbf {S} _ {e 2 t} \left(e ^ {\prime}, t _ {e} ^ {\prime}\right) \right] _ {+},
$$
$$
\mathbf {J} _ {3} = \sum_ {\mathcal {Z}} \sum_ {\mathcal {Z} ^ {\prime}} [ \gamma_ {3} + \mathbf {S} _ {t r t} (t _ {e}, r, t _ {\bar {e}}) - \mathbf {S} _ {t r t} (t _ {e} ^ {\prime}, r, t _ {\bar {e}} ^ {\prime}) ] _ {+}
$$
$\gamma_{1}, \gamma_{2}, \gamma_{3} > 0$  are margin hyperparameters, and the corrupted datasets are built as follows:
$$
\begin{array}{l} \mathcal {D} ^ {\prime} := \{(e ^ {\prime}, r, \tilde {e}) | (e, r, \tilde {e}) \in \mathcal {D}, e ^ {\prime} \in \mathcal {E}, e ^ {\prime} \neq e \} \\ \cup \{(e, r, \tilde {e} ^ {\prime}) | (e, r, \tilde {e}) \in \mathcal {D}, \tilde {e} ^ {\prime} \in \mathcal {E}, \tilde {e} ^ {\prime} \neq \tilde {e} \}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {H} ^ {\prime} := \left\{\left(e ^ {\prime}, t _ {e}\right) \mid \left(e, t _ {e}\right) \in \mathcal {H}, e ^ {\prime} \in \mathcal {E}, e ^ {\prime} \neq e \right\} \\ \cup \{(e, t _ {e} ^ {\prime}) | (e, t _ {e}) \in \mathcal {H}, t _ {e} ^ {\prime} \in \mathcal {T}, t _ {e} ^ {\prime} \neq t _ {e} \}, \\ \mathcal {Z} ^ {\prime} := \left\{\left(t _ {e} ^ {\prime}, r, t _ {\tilde {e}}\right) \mid \left(t _ {e}, r, t _ {\tilde {e}}\right) \in \mathcal {Z}, t _ {e} ^ {\prime} \in \mathcal {T}, t _ {e} ^ {\prime} \neq t _ {e} \right\} \\ \cup \left\{\left(t _ {e}, r, t _ {\bar {e}} ^ {\prime}\right) \mid \left(t _ {e}, r, t _ {\bar {e}}\right) \in \mathcal {Z}, t _ {\bar {e}} ^ {\prime} \in \mathcal {T}, t _ {\bar {e}} ^ {\prime} \neq t _ {\bar {e}} \right\} \\ \end{array}
$$
$\mathcal{D}, \mathcal{H}$  are training datasets of triple facts and entity type instances in KG.  $\mathcal{Z}$  is the training data of type triples, built by replacing entities in  $\mathcal{D}$  with their corresponding entity types.