Knowledge graphs are a vital source for disambiguation and discovery in various tasks such as question answering BIBREF0 , information extraction BIBREF1 and search BIBREF2 . They are, however, known to suffer from data quality issues BIBREF3 . Most prominently, since formal knowledge is inherently sparse, relevant facts are often missing from the graph.To overcome this problem, knowledge graph completion (KGC) or link prediction strives to enrich existing graphs with new facts. Formally, a knowledge graph $\mathcal {G} \, {\subset } \, E {\times } R {\times } E$ consists of facts or triples $(head,rel,tail)$ , where $E$ and $R$ denote finite sets of entities and relations respectively. Knowledge graph completion is targeted at assessing the probability of triples not present in the graph. To do so, a common approach involves representing the entities and relations in triples using real-valued vectors called embeddings. The probability of the triple is then inferred by geometric reasoning over the embeddings. Embeddings are usually generated by learning to discriminate real triples from randomly corrupted ones BIBREF4 , BIBREF5 , BIBREF6 .A key problem with most existing approaches is that the plausibility of links can be determined for known entities only. For many applications, however, it is of interest to infer knowledge about entities not present in the graph. Imagine answering the question “What is German actress Julia Lindig known for?”, where Julia Lindig is not a known entity. Here, information can be inferred from the question, typically using word embeddings BIBREF7 , BIBREF8 , BIBREF9 . Similar to entity embeddings, these techniques represent words with embedding vectors. These can be pre-trained on text corpora, thereby capturing word similarity and semantic relations, which may help to predict the plausibility of the triple $(Julia\_Lindig, starred\_in, Lola\_Rennt)$ . This challenge is known as open-world (or zero-shot) KGC. To the best of our knowledge, few open-world KGC models have been proposed so far, all of which are full replacements for regular KGC models and require textual descriptions for all entities BIBREF10 , BIBREF11 .In this paper, we suggest a different approach, namely to extend existing KGC models with pre-trained word embeddings. Given an new entity, we aggregate its name and description into a text-based entity representation. We then learn a transformation from text-based embedding space to graph-based embedding space, where we can now apply the graph-based model for predicting links. We show that this simple approach yields competitive results, and offers two key benefits: First, it is independent of the specific KGC model used, which allows us to use multiple different link prediction models from which we can pick the best one. Second, as training on the graph structure happens independently from training on text, our approach can exploit the full-scale knowledge graph structure in situations where textual information is scarce because learning the transformation is robust even for such situations. We coin our approach OWE for Open World Extension and combine it with several common KGC models, obtaining TransE-OWE, DistMult-OWE, and ComplEx-OWE.We demonstrate competitive results on common datasets for open-world prediction, and also introduce a new dataset called FB15k-237-OWE, which avoids bias towards long textual descriptions and trivial regularities like inverse relations. The code and the new FB15k-237-OWE dataset are available online.Our approach starts with a regular link prediction model (in the following also referred to as the graph-based model) as outlined in Section "Related Work" and visualised in Fig. 1 . The model scores triples $(h, r, t)$ : $$score(h,r,t) = \phi ({u}_{h}, {u}_{r}, {u}_{t})$$   (Eq. 6) where ${u}_{x}$ denotes the embedding of entity/relation $x$ . Typically, ${u}_{x} \in \mathbb {R}^d$ , but other options are possible. For example, in ComplEx BIBREF6 , ${u}_{x}$ is complex-valued ( ${u}_{x} \in \mathbb {C}^d$ ). $\phi $ is a scoring function that depends on the link prediction model and will be adressed in more detail in Section "Link Prediction Models" .Since our approach is independent of the specific link prediction model used, we test three commonly used models in this work:TransE: $\phi ({u}_{h}, {u}_{r}, {u}_{t}) = -||{u}_{h} {+} {u}_{r} {-} {u}_{t}||_2$ $\;\;\;$ DistMult: $\phi ({u}_{h}, {u}_{r}, {u}_{t}) = \langle {u}_{h},{u}_{r},{u}_{t}\rangle $ $\;\;\;\;\;\;\;\;$ ComplEx: $\phi ({u}_{h}, {u}_{r}, {u}_{t}) = \textrm {Re}({\langle }{u}_{h}, {u}_{r}, {\overline{u}}_{t}{\rangle })$ $\;$ Note that the first two use real-valued embeddings, while ComplEx uses complex-valued embeddings (where ${\overline{u}} = \textrm {Re}({u}) - i{\cdot } \textrm {Im}({u})$ denotes the complex conjugate of embedding ${u}$ ). All models are trained using their original loss functions and validated using closed-world validation data.We use pre-trained word embeddings trained on large text corpora. Since the number of entities in the datasets used is limited and we found overfitting to be an issue, we omit any refinement of the embeddings. We tested 200-dimensional Glove embeddings BIBREF8 and 300-dimensional Wikipedia2Vec embeddings BIBREF18 .Note that Wikipedia2Vec embeddings contain phrase embeddings, which we use as an embedding for entity names (like "Julia Lindig"). If no phrase embedding is available, we split the name into single tokens and use token-wise embeddings. If no embedding is available for a token, we use a vector of zeros as an “unknown” token.To aggregate word embeddings to an entity embedding (function $\Psi ^{agg}$ , Equation 11 ), approaches in the literature range from simple averaging BIBREF8 over Long Short Term Memory Networks (LSTMs) BIBREF23 to relation-specific masking BIBREF11 . We use averaging as an aggregation function. Here, the word embedding vectors are averaged to obtain a single representative embedding. To prevent overfitting, we apply dropout during training, i.e. embeddings of some words are randomly replaced by the unknown token before averaging.The key to open-world prediction is the mapping from text-based entity embeddings ${v}_e$ to graph-based ones ${u}_e$ . Several different transformation functions $\Psi ^{map}$ can be learned for this task. In this paper, we discuss three options:A simple linear function $\Psi ^{map}({v}) = A {\cdot } {v}$ . For ComplEx, separate matrices are used for the real and imaginary part: $\Psi ^{map}({v}) = A {\cdot } {v} + i \cdot A^{\prime } {\cdot } {v}$ Here, $\Psi ^{map}$ is an affine function $\Psi ^{map}({v}) = A {\cdot } {v} + {b}$ . For ComplEx, separate matrices and vectors are trained just like above: $\Psi ^{map}({v}) = (A {\cdot } {v} + {b}) + i \cdot (A^{\prime } {\cdot } {v} + {b^{\prime }})$  $\Psi ^{map}$ is a four layer Multi-Layer Perceptron (MLP) with ReLU activation functions. The output layer is affine. We did not perform an extensive hyperparameter search here.To train the transformations, first a link prediction model is trained on the full graph, obtaining entity embeddings ${u}_1,...,{u}_n$ . We then choose all entities $e_{i_1},...,e_{i_m}$ with textual metadata (names and/or descriptions), and extract text-based embedding ${v}_{i_1},...,{v}_{i_m}$ for them using aggregation (see above). Finally, $\Psi ^{map}$ is learned by minimizing the loss function $$L(\Theta ) = \sum _{k=1}^m \Big | \Big |\Psi _\Theta ^{map}({v}_{i_k}) - {u}_{i_k} \Big |\Big |_2$$   (Eq. 22) using batched stochastic gradient descent, where $\Theta $ denotes the parameters of $\Psi ^{map}$ (e.g., the weight matrices and bias vectors $A,b$ ). For ComplEx, the above loss is summed for real and imaginary parts, and training happens on the sum. We apply no fine-tuning, neither on the graph nor on the text embeddings.In this section, we study the impact of our model's parameters ( $\Psi ^{agg}, \Psi ^{map}$ , text embeddings) on prediction performance. We also provide mappings of selected open-world entities, and compare our results with the state-of-the-art.Closed-world KGC tasks are commonly evaluated on WordNet and Freebase subsets, such as WN18, WN18RR, FB15k, and FB15k-237. For open-world KGC, the following datasets have been suggested: BIBREF10 introduced FB20k, which builds upon the FB15k dataset by adding test triples with unseen entities, which are selected to have long textual descriptions. BIBREF11 introduced DBPedia50k and DBPedia500k datasets for both open-world and closed-world KGC tasks.However, the above datasets display a bias towards long textual descriptions: DBpedia50k has an average description length of 454 words, FB20k of 147 words. Also, for neither of the datasets precautions have been taken to avoid redundant inverse relations, which allows models to exploit trivial patterns in the data BIBREF24 . To overcome these problems, we introduce a new dataset named FB15k-237-OWE. FB15k-237-OWE is based on the well-known FB15K-237 dataset, where redundant inverse relations have been removed. Also, we avoid a bias towards entities with longer textual descriptions: Test entities are uniformly sampled from FB15K-237, and only short Wikidata descriptions (5 words on average) are used.In the following section, the sampling strategy for FB15k-237-OWE is briefly outlined: For tail prediction test set, we start with FB15K-237 and randomly pick heads (by uniform sampling over all head entities). Each picked head $x$ is removed from the training graph by moving all triples of the form $(x,?,t)$ to the test set and dropping all triples of the form $(?,?,x)$ if $t$ still remains in the training set after these operations. Similarly, a head prediction test set is prepared from the set of dropped triplets which satisfy the conditions to be in head prediction test set i.e. head must be represented in training set while tail must not be represented. The dataset also contains two validation sets: A closed-world one (with random triples picked from the training set) and an open-world one (with random triples picked from the test set).We evaluate our approach on DBPedia50k, FB20k, and the new dataset FB15k-237-OWE. Statistics of the datasets are highlighted in Table 1 and Table 2 .We perform multiple runs using different KGC models, transformation types, training data, and embeddings used. For each run, both KGC model and transformation $\Psi ^{map}$ are trained on the training set: the KGC model without using any textual information and the transformation using entity names and descriptions. We manually optimize all hyperparameters on the validation set. Due to the lack of an open-world validation set on FB20k, we randomly sampled 10% of the test triples as a validation set.Performance figures are computed using tail prediction on the test sets: For each test triple $(h,r,t)$ with open-world head $h \notin E$ , we rank all known entities $t^{\prime } \in E$ by their score $\phi (h,r,t^{\prime })$ . We then evaluate the ranks of the target entities $t$ with the commonly used mean rank (MR), mean reciprocal rank (MRR), as well as Hits@1, Hits@3, and Hits@10.Note that multiple triples with the same head and relation but different tails may occur in the dataset: $(h,r,t_1),...,(h,r,t_p)$ . Following BIBREF13 , when evaluating triple $(h,r,t_i)$ we remove all entities $t_1,...,t_{i-1},t_{i+1},...,t_p$ from the result list . All results (except MRR(raw)) are reported with this filtered approach. Note also that when computing the MRR, given a triple $(h,r,t_i)$ only the reciprocal rank of $t_i$ itself is evaluated (and not the best out of $t_1,...,t_{i},...,t_p$ , which would give better results). This is common when evaluating KGC models BIBREF13 but differs from ConMask's evaluation code, which is why one result in Table 3 differs from BIBREF11 (see the (*) mark).Note also that BIBREF11 add a second filtering method called target filtering: When evaluating a test triple $(h,r,t)$ , tails $t^{\prime }$ are only included in the ranked result list if a triple of the form $(?,r,t^{\prime })$ exists in the training data, otherwise it is skipped. We found this to improve quantitative results substantially, but it limits the predictive power of the model because tails can never be linked via new relations. Therefore, we use target filtering only when comparing with the ConMask and DKRL models from BIBREF11 (Table 3 ).For training TransE and DistMult, we use the OpenKE framework which provides implementations of many common link prediction models. For closed-world graph embedding, we use both OpenKE and our own implementation after validating the equivalence of both.For training the transformation $\Psi ^{map}$ , we used the Adam optimizer with a learning rate of $10^{-3}$ and batch size of 128. For DBPedia50k we use a dropout of 0.5, while for FB20k and FB15k-237-OWE we use no dropout. The embedding used is the pretrained 300 dimensional Wikipedia2Vec embedding and the transformation used is affine unless stated otherwise.We first compare our model ComplEx-OWE with other open-world link prediction models in Table 3 . For a fair comparison, all the results are evaluated using target filtering. For all models and all datasets, 200-dimensional Glove embeddings were used, except for the Complex-OWE300, which uses 300-dimensional Wikipedia2Vec embeddings. The effect of different embeddings will be studied further in Section "Text Embeddings and Robustness To Missing Entity Metadata" .The results for Target Filtering Baseline, DKRL and ConMask were obtained by the implementation provided by BIBREF11 . The Target Filtering Baseline is evaluated by assigning random scores to all targets that pass the target filtering criterion. DKRL uses a two-layer CNN over the entity descriptions. ConMask uses a CNN over the entity names and descriptions along with the relation-based attention weights.It can be seen from Table 3 that our best model, ComplEx-OWE300, performs competitively when compared to ConMask. On DBPedia50k, our model performs best on all metrics except Hits@10. On FB20k it is outperformed by a small margin by ConMask but performs better on Hits@1. On FB15k-237-OWE our model outperforms all other models significantly. We believe that this is due to FB15k-237-OWE having very short descriptions. ConMask generally relies on extracting information from the description of entities with its attention mechanism, whereas our model relies more on extracting information from the textual corpus that the word embedding were trained on. This enables our model to provide good results without relying on having long descriptions.Our OWE extension for open-world link prediction can be used with any common KGC model. Therefore, we evaluate three commonly used options, namely TransE, DistMult, and ComplEx. Results are displayed in Table 4 : All three models are trained with embedding dimensionality $d=300$ on the closed-world dataset. For text embeddings, Wikipedia2Vec embeddings of the same dimensionality were used. It can be seen that the performance on the open-world setting matches the expressiveness of the models: ComplEx-OWE with its ability to model even asymmetric relations yields the best results, while the symmetric DistMult-OWE achieves a similar performance.We also test different transformation functions $\Psi ^{map}$ as illustrated in Table 5 . It can be observed that quite simple transformations achieve the strong results: The best performance is achieved by the affine transformation with $49.1$ % HITS@10 by a margin of 2–4 percent.In some cases, the knowledge graph may lack textual metadata (both the name and description) for some or all of its entities. Other models like ConMask and DKRL are dependant on textual descriptions, e.g. ConMask uses attention mechanisms to select relation-specific target words from long texts. Therefore, ConMask and DKRL would require completely dropping triples without metadata and be unable to learn about the link structure of such entities as they use joint training. However, in our approach, we have to drop such entities only during the phase where the transformation $\Psi ^{map}$ is learned, while the link prediction model can still be learned on the full graph.To demonstrate the robustness of our approach to missing entity meta-data, we re-evaluate accuracy when randomly dropping metadata for training entities. Fig. 2 outlines the performance for two scenarios:Dropping descriptions: We remove only the textual descriptions for a varying percentage of randomly selected entities (between 20% to 100%). The names of these entities are not removed and therefore, we still train $\Psi ^{map}$ on them.Dropping all meta-data: We randomly select entities and remove both their descriptions and names, effectively removing these entities from the training set altogether when training $\Psi ^{map}$ .We also included a baseline experiment to simulate an unsuccessful learning of $\Psi ^{map}$ . In this baseline, when evaluating a test triple, we replace its head by the embedding of another random head from the training data. Note that this baseline still gives some reasonable hits for triples where the relation is a strong indicator. For example, if we have a triplet $(X,time\_zone,?)$ : Even if the head $X$ is unknown, a model can achieve reasonable accuracy by simply ”guessing” time zones as tails.Overall, Fig. 2 suggests that transformation learning is able to generalize well even with very limited training data. In Fig. 2 a only the descriptions of entities have been removed. For Wikipedia2Vec embeddings, this removal has virtually no effect on prediction accuracy. We believe that this is because Wikipedia2Vec embeddings are trained such that we can lookup strong entity embeddings by the name alone. Even when removing 100% of descriptions (i.e., only training on the entity names), accuracy is only 2-3% lower than training on the full graph. However, in case of Glove embeddings, the drop in performance is very significant, especially when the description is dropped for all the entities.In Fig. 2 b, we remove not only descriptions but also entity names. Even in this case, learning is robust. If half of the $12,324$ training entities are removed, the drop in accuracy is less than 1%. Only when removing 90% of training data (leaving 123 training entities), performance starts to deteriorate significantly. This highlights the ability of our model to learn from a limited amount of training data, when it is important to be able to train the KGC model itself on all the entities.Finally, we inspect sample prediction results for ComplEx-OWE-300 in Table 6 . Besides the final prediction, we also test whether our transformation from text-based to semantic space is successful: For each test triple, we represent the open-world head entity by its text-based embedding $v_{head}$ , match it to a graph-based embedding $\Psi ^{map}(v_{head})$ , and estimate the nearest neighbor entities in this space. We use the Euclidean distance on the real part of the ComplEx embeddings, but found results to be similar for the imaginary part.If the transformation works well, we expect these nearest neighbors to be semantically similar to the head entity. This is obviously the case: For Bram Stoker (the author of Dracula), the nearest neighbors are other authors of fantasy literature. For Parma, the neighbors are cities (predominantly in Italy). For Bachelor of Science, the model predicts appropriate entities (namely, Universities) but – even though we apply filtering – the predictions are not rewarded. This is because the corresponding triples, like (Bachelor of Science, /.../institution, Harward Law School), are missing in the knowledge graph.In this work, we have presented a simple yet effective extension to embedding-based knowledge graph completion models (such as ComplEx, DistMult and TransE) to perform open-world prediction. Our approach – which we named OWE – maps text-based entity descriptions (learned from word embeddings) to the pre-trained graph embedding space. In experiments on several datasets (including the new FB15K-237-OWE dataset we introduced in this work), we showed that the learned transformations yield semantically meaningful results, that the approach performs competitive with respect to the state of the art, and that it is robust to scarce text descriptions.An interesting direction of future work will be to combine our model with approaches like ConMask BIBREF11 , which (1) exploit more complex aggregation functions and (2) use relation-specific attention/content masking to draw more precise embeddings from longer descriptions.This work was partially funded by the German Federal Ministry of Education and Research (Program FHprofUnt, Project DeepCA / 13FH011PX6) and the German Academic Exchange Service (Project FIBEVID / 57402798).

Q: what was the evaluation metrics studied in this work?