Venue: NAACL 2016

# STransE: a novel embedding model of entities and relationships

in knowledge bases

Dat Quoc Nguyen\({}^{1}\), Kairit Sirts\({}^{1}\), Lizhen Qu\({}^{2}\)  and Mark Johnson\({}^{1}\)

###### Abstract

Knowledge bases of real-world facts about entities and their relationships are useful resources for a variety of natural language processing tasks. However, because knowledge bases are typically incomplete, it is useful to be able to perform _link prediction_, i.e., predict whether a relationship not in the knowledge base is likely to be true. This paper combines insights from several previous link prediction models into a new embedding model _STransE_ that represents each entity as a low-dimensional vector, and each relation by two matrices and a translation vector. STransE is a simple combination of the SE and TransE models, but it obtains better link prediction performance on two benchmark datasets than previous embedding models. Thus, STransE can serve as a new baseline for the more complex models in the link prediction task.

1 Department of Computing, Macquarie University, Sydney, Australia dat.nguyen@students.mq.edu.au, {kairit.sirts, mark.johnson}@mq.edu.au

2 NICTA, ACT 2601, Australia lizhen.qu@nicta.com.au

## 1 Introduction

Knowledge bases (KBs), such as WordNet [3], YAGO [45], Freebase [7] and DBpedia [9], represent relationships between entities as triples \((\mathrm{head~{}entity},\mathrm{relation},\mathrm{tail~{}entity})\). Even very large knowledge bases are still far from complete [28, 29]. _Link prediction_ or _knowledge base completion_ systems [29] predict which triples not in a knowledge base are likely to be true [29, 3]. A variety of different kinds of information is potentially useful here, including information extracted from external corpora [27, 28] and the other relationships that hold between the entities [27, 28]. For example, Toutanova et al. toutanova used information from the external ClueWeb-12 corpus to significantly enhance performance.

While integrating a wide variety of information sources can produce excellent results, there are several reasons for studying simpler models that directly optimize a score function for the triples in a knowledge base, such as the one presented here. First, additional information sources might not be available, e.g., for knowledge bases for specialized domains. Second, models that don't exploit external resources are simpler and thus typically much faster to train than the more complex models using additional information. Third, the more complex models that exploit external information are typically extensions of these simpler models, and are often initialized with parameters estimated by such simpler models, so improvements to the simpler models should yield corresponding improvements to the more complex models as well.

_Embedding models_ for KB completion associate entities and/or relations with dense feature vectors or matrices. Such models obtain state-of-the-art performance [29, 3, 3, 26, 27, 28, 29] and generalize to large KBs [10]. Table 1 summarizes a number of prominent embedding models for KB completion.

Let \((h,r,t)\) represent a triple. In all of the models

discussed here, the head entity \(h\) and the tail entity \(t\) are represented by vectors **h** and \({\textbf{t}}\in\mathbb{R}^{k}\) respectively. The _Unstructured_ model [1] assumes that \({\textbf{h}}\approx{\textbf{t}}\). As the Unstructured model does not take the relationship \(r\) into account, it cannot distinguish different relation types. The _Structured Embedding_ (SE) model [1] extends the unstructured model by assuming that \(h\) and \(t\) are similar only in a relation-dependent subspace. It represents each relation \(r\) with two matrices \({\textbf{W}}_{r,1}\) and \({\textbf{W}}_{r,2}\in\mathbb{R}^{k\times k}\), which are chosen so that \({\textbf{W}}_{r,1}{\textbf{h}}\approx{\textbf{W}}_{r,2}{\textbf{t}}\). The _TransE_ model [1] is inspired by models such as Word2Vec [30] where relationships between words often correspond to translations in latent feature space. The TransE model represents each relation \(r\) by a translation vector \({\textbf{r}}\in\mathbb{R}^{k}\), which is chosen so that \({\textbf{h}}+{\textbf{r}}\approx{\textbf{t}}\).

The primary contribution of this paper is that two very simple relation-prediction models, SE and TransE, can be combined into a single model, which we call _STransE_. Specifically, we use relation-specific matrices \({\textbf{W}}_{r,1}\) and \({\textbf{W}}_{r,2}\) as in the SE model to identify the relation-dependent aspects of both \(h\) and \(t\), and use a vector **r** as in the TransE model to describe the relationship between \(h\) and \(t\) in this subspace. Specifically, our new KB completion model STransE chooses \({\textbf{W}}_{r,1}\), \({\textbf{W}}_{r,2}\) and **r** so that \({\textbf{W}}_{r,1}{\textbf{h}}+{\textbf{r}}\approx{\textbf{W}}_{r,2}{\textbf{ t}}\). That is, a TransE-style relationship holds in some relation-dependent subspace, and crucially, this subspace may involve very different projections of the head \(h\) and tail \(t\). So \({\textbf{W}}_{r,1}\) and \({\textbf{W}}_{r,2}\) can highlight, suppress, or even change the sign of, relation-specific attributes of \(h\) and \(t\). For example, for the "purchases" relationship, certain attributes of individuals \(h\) (e.g., age, gender, marital status) are presumably strongly correlated with very different attributes of objects \(t\) (e.g., sports car, washing machine and the like).

As we show below, STransE performs better than the SE and TransE models and other state-of-the-art link prediction models on two standard link prediction datasets WN18 and FB15k, so it can serve as a new baseline for KB completion. We expect that the STransE will also be able to serve as the basis for extended models that exploit a wider variety of information sources, just as TransE does.

## 2 Our approach

Let \(\mathcal{E}\) denote the set of entities and \(\mathcal{R}\) the set of relation types. For each triple \((h,r,t)\), where \(h,t\in\mathcal{E}\) and \(r\in\mathcal{R}\), the STransE model defines a _score function_\(f_{r}(h,t)\) of its implausibility. Our goal is to choose \(f\) such that the score \(f_{r}(h,t)\) of a plausible triple \((h,r,t)\) is smaller than the score \(f_{r^{\prime}}(h^{\prime},t^{\prime})\) of an implausible triple \((h^{\prime},r^{\prime},t^{\prime})\). We define the STransE score function \(f\) as follows:

\[f_{r}(h,t) = \|{\textbf{W}}_{r,1}{\textbf{h}}+{\textbf{r}}-{\textbf{W}}_{r,2} {\textbf{t}}\|_{\ell_{1/2}}\]

using either the \(\ell_{1}\) or the \(\ell_{2}\)-norm (the choice is made using validation data; in our experiments we found that the \(\ell_{1}\) norm gave slightly better results). To learn the vectors and matrices we minimize the following margin-based objective function: 

\[{\cal L} = \sum_{\begin{subarray}{c}(h,r,t)\in{\cal G}\\ (h^{\prime},r,t^{\prime})\in{\cal G}^{\prime}_{(h,r,t)}\end{subarray}}~{}[ \gamma+f_{r}(h,t)-f_{r}(h^{\prime},t^{\prime})]_{+}\]

where \([x]_{+}=\max(0,x)\), \(\gamma\) is the margin hyper-parameter, \({\cal G}\) is the training set consisting of correct triples, and \({\cal G}^{\prime}_{(h,r,t)}=\{(h^{\prime},r,t)~{}|~{}h^{\prime}\in{\cal E},( h^{\prime},r,t)\notin{\cal G}\}\cup\{(h,r,t^{\prime})~{}|~{}t^{\prime}\in{\cal E},(h, r,t^{\prime})\notin{\cal G}\}\) is the set of incorrect triples generated by corrupting a correct triple \((h,r,t)\in{\cal G}\).

We use Stochastic Gradient Descent (SGD) to minimize \({\cal L}\), and impose the following constraints during training: \(\lVert\textbf{h}\rVert_{2}\leqslant 1\), \(\lVert\textbf{r}\rVert_{2}\leqslant 1\), \(\lVert\textbf{t}\rVert_{2}\leqslant 1\), \(\lVert\textbf{W}_{r,1}\textbf{h}\rVert_{2}\leqslant 1\) and \(\lVert\textbf{W}_{r,2}\textbf{t}\rVert_{2}\leqslant 1\).

## 3 Related work

Table 1 summarizes related embedding models for link prediction and KB completion. The models differ in the score functions \(f_{r}(h,t)\) and the algorithms used to optimize the margin-based objective function, e.g., SGD, AdaGrad Duchi et al. (2011), AdaDelta Zeiler (2012) and L-BFGS Liu and Nocedal (1989).

DISTMULT Yang et al. (2015) is based on a Bilinear model Nickel et al. (2011); Bordes et al. (2012); Jenatton et al. (2012) where each relation is represented by a diagonal rather than a full matrix. The neural tensor network (NTN) model Socher et al. (2013) uses a bilinear tensor operator to represent each relation. Similar quadratic forms are used to model entities and relations in KG2E He et al. (2015) and TATEC Garcia-Duran et al. (2015b).

The TransH model Wang et al. (2014b) associates each relation with a relation-specific hyperplane and uses a projection vector to project entity vectors onto that hyperplane. TransD Ji et al. (2015) and TransR/CTransR Lin et al. (2015b) extend the TransH model using two projection vectors and a matrix to project entity vectors into a relation-specific space, respectively. TransD learns a relation-role specific mapping just as STransE, but represents this mapping by projection vectors rather than full matrices, as in STransE. Thus STransE can be viewed as an extension of the TransR model, where head and tail entities are associated with their own project matrices, rather than using the same matrix for both, as in TransR and CTransR.

Recently, Lao et al. (2011), Neelakantan et al. (2015), Gardner and Mitchell (2015), Luo et al. (2015), Lin et al. (2015a), Garcia-Duran et al. (2015a) and Guu et al. (2015) showed that relation paths between entities in KBs provide richer information and improve the relationship prediction. Nickel et al. (2015) reviews other approaches for learning from KBs and multi-relational data.

## 4 Experiments

For link prediction evaluation, we conduct experiments and compare the performance of our STransE model with published results on the benchmark WN18 and FB15k datasets Bordes et al. (2013). Information about these datasets is given in Table 2.

### Task and evaluation protocol

The link prediction task Bordes et al. (2011); Bordes et al. (2012); Bordes et al. (2013) predicts the head or tail entity given the relation type and the other entity, i.e. predicting \(h\) given \((?,r,t)\) or predicting \(t\) given \((h,r,?)\) where \(?\) denotes the missing element. The results are evaluated using the ranking induced by the score function \(f_{r}(h,t)\) on test triples.

For each test triple \((h,r,t)\), we corrupted it by replacing either \(h\) or \(t\) by each of the possible entities in turn, and then rank these candidates in ascending order of their implausibility value computed by the score function. Following the protocol described in Bordes et al. (2013), we remove any corrupted triples that appear in the knowledge base, to avoid cases where a correct corrupted triple might be ranked higher than the test triple. We report the mean rank and the Hits@10 (i.e., the proportion of test triples in which the target entity was ranked in the top 10 predictions) for each model. Lower mean rank or higher Hits@10 indicates better l

\begin{table}
\begin{tabular}{l|l l l l l} \hline
**Dataset** & **\#E** & **\#R** & **\#Train** & **\#Valid** & **\#Test** \\ \hline WN18 & 40,943 & 18 & 141,442 & 5,000 & 5,000 \\ FB15k & 14,951 & 1,345 & 483,142 & 50,000 & 59,071 \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of the experimental datasets used in this study (and previous works). #E is the number of entities, #R is the number of relation types, and #Train, #Valid and #Test are the numbers of triples in the training, validation and test sets, respectively.

 

Following TransR/CTransR (Lin et al., 2015), TransD (Ji et al., 2015), TATEC (Garcia-Duran et al., 2015b), rTransE (Garcia-Duran et al., 2015a) and PTransE (Lin et al., 2015a), we used the entity and relation vectors produced by TransE (Bordes et al., 2013) to initialize the entity and relation vectors in STransE, and we initialized the relation matrices with identity matrices. Following Wang et al. (2014b), Lin et al. (2015b), He et al. (2015), Ji et al. (2015) and Lin et al. (2015a), we applied the "_Bernoulli_" trick for generating head or tail entities when sampling incorrect triples. We ran SGD for 2,000 epochs to estimate the model parameters. Following Bordes et al. (2013) we used a grid search on validation set to choose either the \(l_{1}\) or \(l_{2}\) norm in the score function \(f\), as well as to set the SGD learning rate \(\lambda\in\{0.0001,0.0005,0.001,0.005,0.01\}\), the margin hyper-parameter \(\gamma\in\{1,3,5\}\) and the number of vector dimensions \(k\in\{50,100\}\). The lowest mean rank on the validation set was obtained when using the \(l_{1}\) norm in \(f\) on both WN18 and FB15k, and when \(\lambda=0.0005,\gamma=5,\text{ and }k=50\) for WN18, and \(\lambda=0.0001,\gamma=1,\text{ and }k=100\) for FB15k.

### Main results

Table 3 compares the link prediction results of our STransE model with results reported in prior work, using the same experimental setup. The first twelve rows report the performance of models that do not exploit information about alternative paths between head and tail entities. The next two rows report results of the rTransE and PTransE models, which are extensions of the TransE model that exploit information about relation paths. The last row presents results for the log-linear model Node+LinkFeat (Toutanova and Chen, 2015) which makes use of textual mentions derived from the large external ClueWeb-12 corpus.

It is clear that Node+LinkFeat with the additional external corpus information obtained best results. In future work we plan to extend the STransE model to incorporate such additional information. Table 3 also shows that models rTransE and PTransE employing path information achieve better results than models that do not use such information. In terms of models not exploiting path information or external information, the STransE model scores better than the other models on WN18 and produces the highest Hits@10 score on FB15k. Compared to the closely related models SE, TransE, TransR, CTransR and TransD, STransE does better than these models on both WN18 and FB15k.

Following Bordes et al. (2013), Table 4 analyzes Hits@10 results on FB15k with respect to the relation categories defined as follows: for each relation type \(r\), we computed the averaged number \(a_{h}\) of heads \(h\) for a pair \((r,t)\) and the averaged number \(a_{t}\) of tails \(t\) for a pair \((h,r)\). If \(a_{h}<1.5\) and \(a_{t}<1.5\), then \(r\) is labeled **1-1**. If \(a_{h}\geq 1.5\) and \(a_{t}<1.5\), then \(r\) is labeled **M-1**. If \(a_{h}<1.5\) and \(a_{t}\geq 1.5\), then \(r\) is labeled as **1-M**. If \(a_{h}\geq 1.5\) and \(a_{t}\geq 1.5\), then \(r\) is labeled as **M-M**. 1.4%, 8.9%, 14.6% and 75.1% of the test triples belong to a relation type classified as **1-1**, **1-M**, **M-1** and **M-M**, respectively.

Table 4 shows that in comparison to prior models not using path information, STransE obtains highest Hits@10 result for **M-M** relation category at \((80.1\%+83.1\%)/2=81.6\%\). In addition, STransE

\begin{table}
\begin{tabular}{p{113.8pt}|c c|c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c|}{**WN18**} & \multicolumn{2}{c}{**FB15k**} \\ \cline{2-5}  & MR & H10 & MR & H10 \\ \hline SE (Bordes et al., 2011) & 985 & 80.5 & 162 & 39.8 \\ Unstructured (Bordes et al., 2012) & 304 & 38.2 & 979 & 6.3 \\ TransE (Bordes et al., 2013) & 251 & 89.2 & 125 & 47.1 \\ TransfH (Wang et al., 2014b) & 303 & 86.7 & 87 & 64.4 \\ TransR (Lin et al., 2015b) & 225 & 92.0 & 77 & 68.7 \\ CTransN (Lin et al., 2015b) & 218 & 92.3 & 75 & 70.2 \\ KG2E (He et al., 2015) & 348 & 93.2 & 59 & 74.0 \\ TransD (Ji et al., 2015) & 212 & 92.2 & 91 & 77.3 \\ TATEC (García-Duran et al., 2015b) & - & - & **58** & 76.7 \\ NTN (Socher et al., 2013) & - & 66.1\({}^{+}\) & - & 41.4\({}^{+}\) \\ DISTMULT (Yang et al., 2015) & - & 94.2\({}^{+}\) & - & 57.7\({}^{+}\) \\ \hline Our STransE model & **206** & **93.4** & 69 & **79.7** \\ \hline \hline \end{tabular} \begin{tabular}{p{113.8pt} p{113.8pt}|p{113.8pt} p{113.8pt}} \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c}{**STransE**} \\ \cline{2-5}  & MR & H10 & MR & H10 \\ \hline SE (Bordes et al., 2011) & 985 & 80.5 & 162 & 39.8 \\ Unstructured (Bordes et al., 2012) & 304 & 38.2 & 979 & 6.3 \\ TransE (Bordes et al., 2013) & 251 & 89.2 & 125 & 47.1 \\ TransH (Wang et al., 2014b) & 303 & 86.7 & 87 & 64.4 \\ TransR (Lin et al., 2015b) & 225 & 92.0 & 77 & 68.7 \\ CTransN (Lin et al., 2015b) & 218 & 92.3 & 75 & 70.2 \\ KG2E (He et al., 2015) & 348 & 93.2 & 59 & 74.0 \\ TransD (Ji et al., 2015) & 212 & 92.2 & 91 & 77.3 \\ TATEC (García-Duran et al., 2015b) & - & - & **58** & 76.7 \\ NTN (Socher et al., 2013) & - & 66.1\({}^{+}\) & - & 41.4\({}^{+}\) \\ DISTMULT (Yang et al., 2015) & - & 94.2\({}^{+}\) & - & 57.7\({}^{+}\) \\ \hline Our STransE model & **206** & **93.4** & 69 & **79.7** \\ \hline \hline \end{tabular} 
\begin{tabular}{p{113.8pt} p{113.8pt}|p{113.8pt} p{113.8pt}} \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c}{**STransE**} \\ \cline{2-5}  & MR & H10 & MR & H10 \\ \hline SE (Bordes et al., 2011) & 985 & 80.5 & 162 & 39.8 \\ Unstructured (Bordes et al., 2012) & 304 & 38.2 & 979 & 6.3 \\ TransE (Bordes et al., 2013) & 251 & 89.2 & 125 & 47.1 \\ TransH (Wang et al., 2014b) & 303 & 86.7 & 87 & 64.4 \\ TransR (Lin et al., 2015b) & 225 & 92.0 & 77 & 68.7 \\ CTransN (Lin et al., 2015b) & 218 & 92.3 & 75 & 70.2 \\ KG2E (He et al., 2015) & 348 & 93.2 & 59 & 74.0 \\ TransD (Ji et al., 2015) & 212 & 92.2 & 91 & 77.3 \\ TATEC (García-Duran et al., 2015b) & - & - & **58** & 76.7 \\ NTN (Socher et al., 2013) & - & 66.1\({}^{+}\) & - & 41.4\({}^{+}\) \\ DISTMULT (Yang et al., 2015) & - & 94.2\({}^{+}\) & - & 57.7\({}^{+}\) \\ \hline Our STransE model & **206** & **93.4** & 69 & **79.7** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Link prediction results. MR and H10 denote evaluation metrics of mean rank and Hits@10 (in %), respectively. “NLFeat” abbreviates Node+LinkFeat. The results for NTN (Socher et al., 2013) listed in this table are taken from Yang et al. (2015) since NTN was originally evaluated on different datasets. The results marked with \({}^{+}\) are obtained using the optimal hyper-parameters chosen to optimize Hits@10 on the validation set; trained in this manner, STransE obtains a mean rank of 244 

also performs better than TransD for **1-M** and **M-1** relation categories. We believe the improved performance of the STransE model is due to its use of full matrices, rather than just projection vectors as in TransD. This permits STransE to model diverse and complex relation categories (such as **1-M**, **M-1** and especially **M-M**) better than TransD and other similiar models. However, STransE is not as good as TransD for the **1-1** relations. Perhaps the extra parameters in STransE hurt performance in this case (note that 1-1 relations are relatively rare, so STransE does better overall).

## 5 Conclusion and future work

This paper presented a new embedding model for link prediction and KB completion. Our STransE combines insights from several simpler embedding models, specifically the Structured Embedding model Bordes et al. (2011) and the TransE model Bordes et al. (2013), by using a low-dimensional vector and two projection matrices to represent each relation. STransE, while being conceptually simple, produces highly competitive results on standard link prediction evaluations, and scores better than the embedding-based models it builds on. Thus it is a suitable candidate for serving as future baseline for more complex models in the link prediction task.

In future work we plan to extend STransE to exploit relation path information in knowledge bases, in a manner similar to Lin et al. (2015a), Garcia-Duran et al. (2015a) or Guu et al. (2015).

## Acknowledgments

This research was supported by a Google award through the Natural Language Understanding Focused Program, and under the Australian Research Council's _Discovery Projects_ funding scheme (project number DP160102156).

NICTA is funded by the Australian Government through the Department of Communications and the Australian Research Council through the ICT Centre of Excellence Program. The first author is supported by an International Postgraduate Research Scholarship and a NICTA NRPA Top-Up Scholarship.

## References

* Angeli and Manning (2013) Gabor Angeli and Christopher Manning. 2013. Philosophers are Mortal: Inferring the Truth of Unseen Facts. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_, pages 133-142.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_, pages 1247-1250.
* Bordes et al. (2011) Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning Structured Embeddings of Knowledge Bases. In _Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence_, pages 301-306.
* Bordes et al. (2012) Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. A Semantic Matching Energy Function for Learning with Multi-relational Data. _Machine Learning_, 94(2):233-259.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In _Advances in Neural Information Processing Systems 26_, pages 2787-2795.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. _The Journal of Machine Learning Research_, 12:2121-2159.
* Fellbaum (1998) Christiane D. Fellbaum. 1998. _WordNet: An Electronic Lexical Database_. MIT Press.
* Garcia-Duran et al. (2015a) Alberto Garcia-Duran, Antoine Bordes, and Nicolas Usunier. 2015a. Composing Relationships with Translations. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 286-290.
* Garcia-Duran et al. (2015b)

\begin{table}
\begin{tabular}{l|c c c c|c c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{5}{c|}{**Predicting head**\(h\)} & \multicolumn{5}{c}{**Predicting tail**\(t\)} \\ \cline{2-9}  & 1-1 & 1-M & M-1 & M-M & 1-1 & 1-M & M-1 & M-M \\ \hline SE & 35.6 & 62.6 & 17.2 & 37.5 & 34.9 & 14.6 & 68.3 & 41.3 \\ Unstr. & 34.5 & 2.5 & 6.1 & 6.6 & 34.3 & 4.2 & 1.9 & 6.6 \\ TransE & 43.7 & 65.7 & 18.2 & 47.2 & 43.7 & 19.7 & 66.7 & 50.0 \\ TransH & 66.8 & 87.6 & 28.7 & 64.5 & 65.5 & 39.8 & 83.3 & 67.2 \\ TransR & 78.8 & 89.2 & 34.1 & 69.2 & 79.2 & 37.4 & 90.4 & 72.1 \\ CTransR & 81.5 & 89.0 & 34.7 & 71.2 & 80.8 & 38.6 & 90.1 & 73.8 \\ KG2E & **92.3** & 94.6 & **66.0** & 69.6 & **92.6** & **67.9** & **94.4** & 73.4 \\ TransD & 86.1 & **95.5** & 39.8 & 78.5 & 85.4 & 50.6 & **94.4** & 81.2 \\ TATEC & 79.3 & 93.2 & 42.3 & 77.2 & 78.5 & 51.5 & 92.7 & 80.7 \\ \hline STransE & 82.8 & 94.2 & 50.4 & **80.1** & 82.4 & 56.9 & 93.4 & **83.1** \\ \end{tabular}
\end{table}
Table 4: Hits@10 (in %) by the relation category on FB15k. “Unstr.” abbreviates Unstructured.

 

Alberto Garcia-Duran, Antoine Bordes, Nicolas Usunier, and Yves Grandvalet. 2015b. Combining Two And Three-Way Embeddings Models for Link Prediction in Knowledge Bases. _CoRR_, abs/1506.00999.
* Gardner and Mitchell (2015) Matt Gardner and Tom Mitchell. 2015. Efficient and Expressive Knowledge Base Completion Using Subgraph Feature Extraction. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1488-1498.
* Guu et al. (2015) Kelvin Guu, John Miller, and Percy Liang. 2015. Traversing Knowledge Graphs in Vector Space. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 318-327.
* He et al. (2015) Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao. 2015. Learning to Represent Knowledge Graphs with Gaussian Embedding. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_, pages 623-632.
* Jenatton et al. (2012) Rodolphe Jenatton, Nicolas L. Roux, Antoine Bordes, and Guillaume R Obozinski. 2012. A latent factor model for highly multi-relational data. In _Advances in Neural Information Processing Systems 25_, pages 3167-3175.
* Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 687-696.
* Krompa et al. (2015) Denis Krompa, Stephan Baier, and Volker Tresp. 2015. Type-Constrained Representation Learning in Knowledge Graphs. In _Proceedings of the 14th International Semantic Web Conference_, pages 640-655.
* Lao et al. (2011) Ni Lao, Tom Mitchell, and William W. Cohen. 2011. Random Walk Inference and Learning in a Large Scale Knowledge Base. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 529-539.
* A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia. _Semantic Web_, 6(2):167-195.
* Lin et al. (2015a) Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015a. Modeling Relation Paths for Representation Learning of Knowledge Bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 705-714.
* Lin et al. (2015b) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Learning_, pages 2181-2187.
* Liu and Nocedal (1989) D. C. Liu and J. Nocedal. 1989. On the Limited Memory BFGS Method for Large Scale Optimization. _Mathematical Programming_, 45(3):503-528.
* Luo et al. (2015) Yuanfei Luo, Quan Wang, Bin Wang, and Li Guo. 2015. Context-Dependent Knowledge Graph Embedding. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1656-1661.
* Mikolov et al. (2013) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 746-751.
* Neelakantan et al. (2015) Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. 2015. Compositional Vector Space Models for Knowledge Base Completion. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 156-166.
* Nickel et al. (2011) Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A Three-Way Model for Collective Learning on Multi-Relational Data. In _Proceedings of the 28th International Conference on Machine Learning_, pages 809-816.
* Nickel et al. (2015) Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A Review of Relational Machine Learning for Knowledge Graphs. _Proceedings of the IEEE, to appear_.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation Extraction with Matrix Factorization and Universal Schemas. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 74-84.
* Socher et al. (2013) Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In _Advances in Neural Information Processing Systems 26_, pages 926-934.
* Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. YAGO: A Core of Semantic Knowledge. In _Proceedings of the 16th International Conference on World Wide Web_, pages 697-706.
* Taskar et al. (2004) Ben Taskar, Ming fai Wong, Pieter Abbeel, and Daphne Koller. 2004. Link Prediction in Relational Data. In 

Advances in Neural Information Processing Systems 16_, pages 659-666.
* Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015. Observed Versus Latent Features for Knowledge Base and Text Inference. In _Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality_, pages 57-66.
* Toutanova et al. (2015) Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing Text for Joint Embedding of Text and Knowledge Bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1499-1509.
* Wang et al. (2014a) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014a. Knowledge Graph and Text Jointly Embedding. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1591-1601.
* Wang et al. (2014b) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014b. Knowledge Graph Embedding by Translating on Hyperplanes. In _Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence_, pages 1112-1119.
* West et al. (2014) Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge Base Completion via Search-based Question Answering. In _Proceedings of the 23rd International Conference on World Wide Web_, pages 515-526.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In _Proceedings of the International Conference on Learning Representations_.
* Zeiler (2012) Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. _CoRR_, abs/1212.5701.
* Zhao et al. (2015) Yu Zhao, Sheng Gao, Patrick Gallinari, and Jun Guo. 2015. Knowledge Base Completion by Learning Pairwise-Interaction Differentiated Embeddings. _Data Mining and Knowledge Discovery_, 29(5):1486-1504.