Introduction
Before introducing the KB completion task in details, let us return to the classic Word2Vec example of a “royal” relationship between “ $\mathsf {king}$ ” and “ $\mathsf {man}$ ”, and between “ $\mathsf {queen}$ ” and “ $\mathsf {woman}$ .” As illustrated in this example: $v_{king} - v_{man} \approx v_{queen} - v_{woman}$ , word vectors learned from a large corpus can model relational similarities or linguistic regularities between pairs of words as translations in the projected vector space BIBREF0 , BIBREF1 . Figure 1 shows another example of a relational similarity between word pairs of countries and capital cities: $ v_{Japan} - v_{Tokyo} &\approx & v_{Germany} - v_{Berlin}\\ v_{Germany} - v_{Berlin} &\approx & v_{Italy} - v_{Rome} \\ v_{Italy} - v_{Rome} &\approx & v_{Portugal} - v_{Lisbon} $
Let us consider the country and capital pairs in Figure 1 to be pairs of entities rather than word types. That is, we now represent country and capital entities by low-dimensional and dense vectors. The relational similarity between word pairs is presumably to capture a “ $\mathsf {is\_capital\_of}$ ” relationship between country and capital entities. Also, we represent this relationship by a translation vector $v_{{is\_capital\_of}}$ in the entity vector space. Thus, we expect: $ v_{Tokyo} + v_{{is\_capital\_of}} - v_{Japan} &\approx & 0 \\ v_{Berlin} + v_{{is\_capital\_of}} - v_{Germany} &\approx & 0 \\ v_{Rome} + v_{{is\_capital\_of}} - v_{Italy} &\approx & 0 \\ v_{Lisbon} + v_{{is\_capital\_of}} - v_{Portugal} &\approx & 0 $
This intuition inspired the TransE model—a well-known embedding model for KB completion or link prediction in KBs BIBREF2 .
Knowledge bases are collections of real-world triples, where each triple or fact $(h, r, t)$ in KBs represents some relation $r$ between a head entity $h$ and a tail entity $t$ . KBs can thus be formalized as directed multi-relational graphs, where nodes correspond to entities and edges linking the nodes encode various kinds of relationship BIBREF3 , BIBREF4 . Here entities are real-world things or objects such as persons, places, organizations, music tracks or movies. Each relation type defines a certain relationship between entities. For example, as illustrated in Figure 2 , the relation type “ $\mathsf {child\_of}$ ” relates person entities with each other, while the relation type “ $\mathsf {born\_in}$ ” relates person entities with place entities. Several KB examples include the domain-specific KB GeneOntology and popular generic KBs of WordNet BIBREF5 , YAGO BIBREF6 , Freebase BIBREF7 , NELL BIBREF8 and DBpedia BIBREF9 as well as commercial KBs such as Google's Knowledge Graph, Microsoft's Satori and Facebook's Open Graph. Nowadays, KBs are used in a number of commercial applications including search engines such as Google, Microsoft's Bing and Facebook's Graph search. They also are useful resources for many NLP tasks such as question answering BIBREF10 , BIBREF11 , word sense disambiguation BIBREF12 , BIBREF13 , semantic parsing BIBREF14 , BIBREF15 and co-reference resolution BIBREF16 , BIBREF17 .
A main issue is that even very large KBs, such as Freebase and DBpedia, which contain billions of fact triples about the world, are still far from complete. In particular, in English DBpedia 2014, 60% of person entities miss a place of birth and 58% of the scientists do not have a fact about what they are known for BIBREF19 . In Freebase, 71% of 3 million person entities miss a place of birth, 75% do not have a nationality while 94% have no facts about their parents BIBREF20 . So, in terms of a specific application, question answering systems based on incomplete KBs would not provide a correct answer given a correctly interpreted question. For example, given the incomplete KB in Figure 2 , it would be impossible to answer the question “where was Jane born ?”, although the question is completely matched with existing entity and relation type information (i.e., “ $\mathsf {Jane}$ ” and “ $\mathsf {born\_in}$ ”) in KB. Consequently, much work has been devoted towards knowledge base completion to perform link prediction in KBs, which attempts to predict whether a relationship/triple not in the KB is likely to be true, i.e., to add new triples by leveraging existing triples in the KB BIBREF21 , BIBREF22 , BIBREF23 , BIBREF3 . For example, we would like to predict the missing tail entity in the incomplete triple $\mathsf {(Jane, born\_in, ?)}$ or predict whether the triple $\mathsf {(Jane, born\_in, Miami)}$ is correct or not.
Embedding models for KB completion have been proven to give state-of-the-art link prediction performances, in which entities are represented by latent feature vectors while relation types are represented by latent feature vectors and/or matrices and/or third-order tensors BIBREF24 , BIBREF25 , BIBREF2 , BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF19 , BIBREF30 , BIBREF3 , BIBREF31 , BIBREF32 , BIBREF33 . This article briefly overviews the embedding models for KB completion, and then summarizes up-to-date experimental results on two standard evaluation tasks: i) the entity prediction task—which is also referred to as the link prediction task BIBREF2 —and ii) the triple classification task BIBREF34 .
A general approach
Let $\mathcal {E}$ denote the set of entities and $\mathcal {R}$ the set of relation types. Denote by $\mathcal {G}$ the knowledge base consisting of a set of correct triples $(h, r, t)$ , such that $h, t \in \mathcal {E}$ and $r \in \mathcal {R}$ . For each triple $(h, r, t)$ , the embedding models define a score function $f(h, r, t)$ of its implausibility. Their goal is to choose $f$ such that the score $f(h, r, t)$ of a plausible triple $\mathcal {R}$0 is smaller than the score $\mathcal {R}$1 of an implausible triple $\mathcal {R}$2 .
Table 1 summarizes different score functions $f(h, r, t)$ and the optimization algorithms used to estimate model parameters. To learn model parameters (i.e., entity vectors, relation vectors or matrices), the embedding models minimize an objective function. A common objective function is the following margin-based function:
$$\mathcal {L} & = \sum _{\begin{array}{c}(h,r,t) \in \mathcal {G} \\ (h^{\prime },r,t^{\prime }) \in \mathcal {G}^{\prime }_{(h, r, t)}\end{array}} [\gamma + f(h, r, t) - f(h^{\prime }, r, t^{\prime })]_+ \nonumber $$   (Eq. 5)
where $[x]_+ = \max (0, x)$ , $\gamma $ is the margin hyper-parameter, and $\mathcal {G}^{\prime }_{(h, r, t)} $ is the set of incorrect triples generated by corrupting the correct triple $(h, r, t)\in \mathcal {G}$ .
Specific models
The Unstructured model BIBREF22 assumes that the head and tail entity vectors are similar. As the Unstructured model does not take the relationship into account, it cannot distinguish different relation types. The Structured Embedding (SE) model BIBREF35 assumes that the head and tail entities are similar only in a relation-dependent subspace, where each relation is represented by two different matrices. Furthermore, the SME model BIBREF22 uses four different matrices to project entity and relation vectors into a subspace. The TransE model BIBREF2 is inspired by models such as the Word2Vec Skip-gram model BIBREF0 where relationships between words often correspond to translations in latent feature space. TorusE BIBREF36 embeds entities and relations on a torus to handle TransE's regularization problem.
The TransH model BIBREF26 associates each relation with a relation-specific hyperplane and uses a projection vector to project entity vectors onto that hyperplane. TransD BIBREF37 and TransR/CTransR BIBREF28 extend the TransH model by using two projection vectors and a matrix to project entity vectors into a relation-specific space, respectively. Similar to TransR, TransR-FT BIBREF38 also uses a matrix to project head and tail entity vectors. TEKE_H BIBREF39 extends TransH to incorporate rich context information in an external text corpus. lppTransD BIBREF40 extends TransD to additionally use two projection vectors for representing each relation. STransE BIBREF41 and TranSparse BIBREF42 can be viewed as direct extensions of the TransR model, where head and tail entities are associated with their own projection matrices. Unlike STransE, the TranSparse model uses adaptive sparse matrices, whose sparse degrees are defined based on the number of entities linked by relations. TranSparse-DT BIBREF43 is an extension of TranSparse with a dynamic translation. ITransF BIBREF44 can be considered as a generalization of STransE, which allows sharing statistic regularities between relation projection matrices and alleviates data sparsity issue.
DISTMULT BIBREF45 is based on the Bilinear model BIBREF24 , BIBREF22 , BIBREF25 where each relation is represented by a diagonal rather than a full matrix. The neural tensor network (NTN) model BIBREF34 uses a bilinear tensor operator to represent each relation while ER-MLP BIBREF27 and ProjE BIBREF46 can be viewed as simplified versions of NTN. Such quadratic forms are also used to model entities and relations in KG2E BIBREF47 , TransG BIBREF48 , ComplEx BIBREF31 , TATEC BIBREF3 , RSTE BIBREF49 and ANALOGY BIBREF50 . In addition, the HolE model BIBREF33 uses circular correlation–a compositional operator–which can be interpreted as a compression of the tensor product.
ConvE BIBREF51 and ConvKB BIBREF52 are based on convolutional neural networks. ConvE uses a 2D convolutional layer directly over head-entity and relation vector embeddings while ConvKB applies a convolutional layer over embedding triples. Unlike ConvE and ConvKB, the IRN model BIBREF53 uses a shared memory and recurrent neural network-based controller to implicitly model multi-step structured relationships.
Recent research has shown that relation paths between entities in KBs provide richer context information and improve the performance of embedding models for KB completion BIBREF54 , BIBREF55 , BIBREF56 , BIBREF29 , BIBREF32 , BIBREF57 , BIBREF58 . BIBREF54 constructed relation paths between entities and, viewing entities and relations in the path as pseudo-words, then applied Word2Vec algorithms BIBREF0 to produce pre-trained vectors for these pseudo-words. BIBREF54 showed that using these pre-trained vectors for initialization helps to improve the performance of models TransE BIBREF2 , SME BIBREF22 and SE BIBREF35 . BIBREF55 used the implausibility score produced by SME to compute the weights of relation paths.
PTransE-RNN BIBREF59 models relation paths by using a recurrent neural network. In addition, rTransE BIBREF56 , PTransE-ADD BIBREF59 and TransE-comp BIBREF29 are extensions of the TransE model. These models similarly represent a relation path by a vector which is the sum of the vectors of all relations in the path, whereas in the Bilinear-comp model BIBREF29 and the pruned-paths model BIBREF32 , each relation is a matrix and so it represents the relation path by matrix multiplication. The neighborhood mixture model TransE-NMM BIBREF57 can be also viewed as a three-relation path model as it takes into account the neighborhood entity and relation information of both head and tail entities in each triple. Neighborhood information is also exploited in the relational graph convolutional networks R-GCN BIBREF60 . Furthermore, BIBREF58 proposed the KB $_{LRN}$ framework to combine relational paths of length one and two with latent and numerical features.
Other KB completion models
The Path Ranking Algorithm (PRA) BIBREF21 is a random walk inference technique which was proposed to predict a new relationship between two entities in KBs. BIBREF61 used PRA to estimate the probability of an unseen triple as a combination of weighted random walks that follow different paths linking the head entity and tail entity in the KB. BIBREF23 made use of an external text corpus to increase the connectivity of the KB used as the input to PRA. BIBREF62 improved PRA by proposing a subgraph feature extraction technique to make the generation of random walks in KBs more efficient and expressive, while BIBREF63 extended PRA to couple the path ranking of multiple relations. PRA can also be used in conjunction with first-order logic in the discriminative Gaifman model BIBREF64 . In addition, BIBREF65 used a recurrent neural network to learn vector representations of PRA-style relation paths between entities in the KB. Other random-walk based learning algorithms for KB completion can be also found in BIBREF66 , BIBREF67 , BIBREF68 and BIBREF69 . Recently, BIBREF70 have proposed a Neural Logic Programming (LP) framework to learning probabilistic first-order logical rules for KB reasoning, producing competitive link prediction performances. See other methods for learning from KBs and multi-relational data in BIBREF4 .
Evaluation tasks
Two standard tasks are proposed to evaluate embedding models for KB completion including: the entity prediction task, i.e. link prediction BIBREF2 , and the triple classification task BIBREF34 .
Information about benchmark datasets for KB completion evaluation is given in Table 2 . Commonly, datasets FB15k and WN18 BIBREF2 are used for entity prediction evaluation, while datasets FB13 and WN11 BIBREF34 are used for triple classification evaluation. FB15k and FB13 are derived from the large real-world fact KB FreeBase BIBREF7 . WN18 and WN11 are derived from the large lexical KB WordNet BIBREF71 .
BIBREF30 noted that FB15k and WN18 are not challenging datasets because they contain many reversible triples. BIBREF51 showed a concrete example: A test triple ( $\mathsf {feline, hyponym, cat}$ ) can be mapped to a training triple ( $\mathsf {cat, hypernym, feline}$ ), thus knowing that “ $\mathsf {hyponym}$ ” and “ $\mathsf {hypernym}$ ” are reversible allows us to easily predict the majority of test triples. So, datasets FB15k-237 BIBREF30 and WN18RR BIBREF51 are created to serve as realistic KB completion datasets which represent a more challenging learning setting. FB15k-237 and WN18RR are subsets of FB15k and WN18, respectively. Note that when creating the FB13 and WN11 datasets, BIBREF34 already filtered out triples from the test set if either or both of their head and tail entities also appear in the training set in a different relation type or order.
Entity prediction
The entity prediction task, i.e. link prediction BIBREF2 , predicts the head or the tail entity given the relation type and the other entity, i.e. predicting $h$ given $(?, r, t)$ or predicting $t$ given $(h, r, ?)$ where $?$ denotes the missing element. The results are evaluated using a ranking induced by the function $f(h, r, t)$ on test triples. Each correct test triple $(h, r, t)$ is corrupted by replacing either its head or tail entity by each of the possible entities in turn, and then these candidates are ranked in ascending order of their implausibility score. This is called as the “Raw” setting protocol. Furthermore, the “Filtered” setting protocol, described in NIPS20135071, filters out before ranking any corrupted triples that appear in the KB. Ranking a corrupted triple appearing in the KB (i.e. a correct triple) higher than the original test triple is also correct, but is penalized by the “Raw” score, thus the “Filtered” setting provides a clearer view on the ranking performance.
In addition to the mean rank and the Hits@10 (i.e., the proportion of test triples for which the target entity was ranked in the top 10 predictions), which were originally used in the entity prediction task BIBREF2 , recent work also reports the mean reciprocal rank (MRR). In both “Raw” and “Filtered” settings, mean rank is always greater or equal to 1 and the lower mean rank indicates better entity prediction performance. MRR and Hits@10 scores always range from 0.0 to 1.0, and higher score reflects better prediction result.
Table 3 lists entity prediction results of KB completion models on the FB15k and WN18 datasets. The first 26 rows report the performance of triple-based models that directly optimize a score function for the triples in a KB, i.e. they do not exploit information about alternative paths between head and tail entities. The next 9 rows report results of models that exploit information about relation paths. The last 3 rows present results for models which make use of textual mentions derived from a large external corpus. The reasons why much work has been devoted towards developing triple-based models are mentioned by BIBREF41 as follows: (1) additional information sources might not be available, e.g., for KBs for specialized domains, (2) models that do not exploit path information or external resources are simpler and thus typically much faster to train than the more complex models using path or external information, and (3) the more complex models that exploit path or external information are typically extensions of these simpler models, and are often initialized with parameters estimated by such simpler models, so improvements to the simpler models should yield corresponding improvements to the more complex models as well.
Table 3 shows that the models using external corpus information or employing path information generally achieve better scores than the triple-based models that do not use such information. In terms of models not exploiting path or external information, on FB15k the IRN model BIBREF53 obtains highest scores, followed by DISTMULT BIBREF45 , ProjE BIBREF46 and ConvE BIBREF51 . On WN18 top-4 triple-based models are ConvE, IRN, TorusE BIBREF36 and ANALOGY BIBREF50 .
Table 4 lists recent results on datasets FB15k-237 and WN18RR. On FB15k-237, by exploiting external textual mentions of entities, the Conv-E + Conv-DISTMULT model BIBREF75 produces the highest Hits@10 and MRR. In terms of models not exploiting external textual information, on FB15k-237, ER-MLP BIBREF27 can be considered as the best model to date, followed by ConvKB BIBREF52 and KB $_{LRN}$ BIBREF58 . On WN18RR, ConvKB can be considered as the best one, followed by ComplEx BIBREF31 and TransE BIBREF2 . Clearly, tables 3 and 4 show that TransE, despite of its simplicity, can produce very competitive results (by performing a careful grid search of hyper-parameters).
Triple classification
The triple classification task was first introduced by NIPS20135028, and since then it has been used to evaluate various embedding models. The aim of this task is to predict whether a triple $(h, r, t)$ is correct or not. For classification, a relation-specific threshold $\theta _r$ is set for each relation type $r$ . If the implausibility score of an unseen test triple $(h, r, t)$ is smaller than $\theta _r$ then the triple will be classified as correct, otherwise incorrect. Following NIPS20135028, the relation-specific thresholds are determined by maximizing the micro-averaged accuracy, which is a per-triple average, on the validation set.
Table 5 presents the triple classification results of KB completion models on the WN11 and FB13 datasets. The first 6 rows report the performance of models that use TransE to initialize the entity and relation vectors. The last 12 rows present the accuracy of models with randomly initialized parameters. Note that there are higher results reported for NTN, Bilinear-comp and TransE-comp when entity vectors are initialized by averaging the pre-trained word vectors BIBREF0 , BIBREF1 . It is not surprising as many entity names in WordNet and FreeBase are lexically meaningful. It is possible for all other embedding models to utilize the pre-trained word vectors as well. However, as pointed out by BIBREF26 and BIBREF29 , averaging the pre-trained word vectors for initializing entity vectors is an open problem and it is not always useful since entity names in many domain-specific KBs are not lexically meaningful.
Conclusions and further discussion
This article presented a brief overview of embedding models of entity and relationships for KB completion. The article also provided update-to-date experimental results of the embedding models on the entity prediction and triple classification tasks on benchmark datasets FB15k, WN18, FB15k-237, WN18RR, FB13 and WN11.
Dozens of embedding models have been proposed for KB completion, so it is worth to further explore these models for a new application where we could formulate its corresponding data into triples. For example of an interesting application, BIBREF76 extended the STransE model BIBREF41 for a search personalization task in information retrieval, to model user-oriented relationships between submitted queries and documents returned by search engines.

Question: What models does this overview cover?
Answer: This article presented a brief overview of embedding models of entity and relationships for KB completion.