Venue: NAACL 2018

# A Novel Embedding Model for Knowledge Base Completion

Based on Convolutional Neural Network

Dai Quoc Nguyen\({}^{1}\), Tu Dinh Nguyen\({}^{1}\), Dat Quoc Nguyen\({}^{2}\), Dinh Phung\({}^{1}\)

\({}^{1}\) Deakin University, Geelong, Australia, Centre for Pattern Recognition and Data Analytics

{dai.nguyen,tu.nguyen,dinh.phung}@deakin.edu.au

\({}^{2}\) The University of Melbourne, Australia

dqnguyen@unimelb.edu.au

###### Abstract

In this paper, we propose a novel embedding model, named ConvKB, for knowledge base completion. Our model ConvKB advances state-of-the-art models by employing a convolutional neural network, so that it can capture global relationships and transitional characteristics between entities and relations in knowledge bases. In ConvKB, each triple _(head entity, relation, tail entity)_ is represented as a 3-column matrix where each column vector represents a triple element. This 3-column matrix is then fed to a convolution layer where multiple filters are operated on the matrix to generate different feature maps. These feature maps are then concatenated into a single feature vector representing the input triple. The feature vector is multiplied with a weight vector via a dot product to return a score. This score is then used to predict whether the triple is valid or not. Experiments show that ConvKB achieves better link prediction performance than previous state-of-the-art embedding models on two benchmark datasets WN18RR and FB15k-237.

## 1 Introduction

Large-scale knowledge bases (KBs), such as YAGO Suchanek et al. (2007), Freebase Bollacker et al. (2008) and DBpedia Lehmann et al. (2015), are usually databases of triples representing the relationships between entities in the form of fact _(head entity, relation, tail entity)_ denoted as _(h, r, t)_, e.g., _(Melbourne, cityOf, Australia)_. These KBs are useful resources in many applications such as semantic searching and ranking Kasneci et al. (2008); Schuhmacher and Ponzetto (2014); Xiong et al. (2017), question answering Zhang et al. (2016); Hao et al. (2017) and machine reading Yang Mitchell (2017). However, the KBs are still incomplete, i.e., missing a lot of valid triples Socher et al. (2013); West et al. (2014). Therefore, much research work has been devoted towards _knowledge base completion_ or _link prediction_ to predict whether a triple _(h, r, t)_ is valid or not Bordes et al. (2011).

Many embedding models have proposed to learn vector or matrix representations for entities and relations, obtaining state-of-the-art (SOTA) link prediction results Nickel et al. (2016a). In these embedding models, valid triples obtain lower implausibility scores than invalid triples. Let us take the well-known embedding model TransE Bordes et al. (2013) as an example. In TransE, entities and relations are represented by \(k\)-dimensional vector embeddings. TransE employs a transitional characteristic to model relationships between entities, in which it assumes that if _(h, r, t)_ is a valid fact, the embedding of head entity \(h\) plus the embedding of relation \(r\) should be close to the embedding of tail entity \(t\), i.e. \(\mathbf{v}_{h}+\mathbf{v}_{r}\approx\mathbf{v}_{t}\) (here, \(\mathbf{v}_{h}\), \(\mathbf{v}_{r}\) and \(\mathbf{v}_{t}\) are embeddings of \(h\), \(r\) and \(t\) respectively). That is, a TransE score \(\|\mathbf{v}_{h}+\mathbf{v}_{r}-\mathbf{v}_{t}\|_{p}\) of the valid triple _(h, r, t)_ should be close to \(0\) and smaller than a score \(\|\mathbf{v}_{h^{\prime}}+\mathbf{v}_{r^{\prime}}-\mathbf{v}_{tr}\|_{p}\) of an invalid triple _(h', r', t')_. The transitional characteristic in TransE also implies the global relationships among same dimensional entries of \(\mathbf{v}_{h}\), \(\mathbf{v}_{r}\) and \(\mathbf{v}_{t}\).

Other transition-based models extend TransE to additionally use projection vectors or matrices to translate head and tail embeddings into the relation vector space, such as: TransH Wang et al. (2014), TransR Lin et al. (2015b), TransD Ji et al. (2015), STransE Nguyen et al. (2016b) and TranSparse Ji et al. (2016). Furthermore, DISTMULT Yang et al. (2015) and ComplEx Trouillon et al. (2016) use a tri-linear dot product to compute the score for each triple. Recent research

has shown that using relation paths between entities in the KBs could help to get contextual information for improving KB completion performance (Lin et al., 2015; Luo et al., 2015; Guu et al., 2015; Toutanova et al., 2016; Nguyen et al., 2016a). See other embedding models for KB completion in Nguyen (2017).

Recently, convolutional neural networks (CNNs), originally designed for computer vision LeCun et al. (1998), have significantly received research attention in natural language processing Collobert et al. (2011); Kim (2014). CNN learns non-linear features to capture complex relationships with a remarkably less number of parameters compared to fully connected neural networks. Inspired from the success in computer vision, Dettmers et al. (2018) proposed ConvE--the first model applying CNN for the KB completion task. In ConvE, only \({\mathbf{v}_{h}}\) and \({\mathbf{v}_{r}}\) are reshaped and then concatenated into an input matrix which is fed to the convolution layer. Different filters of the same \(3\times 3\) shape are operated over the input matrix to output feature map tensors. These feature map tensors are then vectorized and mapped into a vector via a linear transformation. Then this vector is computed with \({\mathbf{v}_{t}}\) via a dot product to return a score for _(h, r, t)_. See a formal definition of the ConvE score function in Table 1. It is worth noting that ConvE focuses on the local relationships among different dimensional entries in each of \({\mathbf{v}_{h}}\) or \({\mathbf{v}_{r}}\), i.e., _ConvE does not observe the global relationships among same dimensional entries of an embedding triple (\({\mathbf{v}_{h}}\), \({\mathbf{v}_{r}}\), \({\mathbf{v}_{t}}\))_, so that ConvE ignores the transitional characteristic in transition-based models, which is one of the most useful intuitions for the task.

In this paper, we present ConvKB--an embedding model which proposes a novel use of CNN for the KB completion task. In ConvKB, each entity or relation is associated with an unique \(k\)-dimensional embedding. Let \({\mathbf{v}_{h}}\), \({\mathbf{v}_{r}}\) and \({\mathbf{v}_{t}}\) denote \(k\)-dimensional embeddings of \(h\), \(r\) and \(t\), respectively. For each triple _(h, r, t)_, the corresponding triple of \(k\)-dimensional embeddings (\({\mathbf{v}_{h}}\), \({\mathbf{v}_{r}}\), \({\mathbf{v}_{t}}\)) is represented as a \(k\times 3\) input matrix. This input matrix is fed to the convolution layer where different filters of the same \(1\times 3\) shape are used to extract the global relationships among same dimensional entries of the embedding triple. That is, these filters are repeatedly operated over every row of the input matrix to produce different feature maps. The feature maps are concatenated into a single feature vector which is then computed with a weight vector via a dot product to produce a score for the triple _(h, r, t)_. This score is used to infer whether the triple _(h, r, t)_ is valid or not.

Our contributions in this paper are as follows:

* We introduce ConvKB--a novel embedding model of entities and relationships for knowledge base completion. ConvKB models the relationships among same dimensional entries of the embeddings. This implies that ConvKB generalizes transitional characteristics in transition-based embedding models.
* We evaluate ConvKB on two benchmark datasets: WN18RR Dettmers et al. (2018) and FB15k-237 Toutanova and Chen (2015). Experimental results show that ConvKB obtains better link prediction performance than previous SOTA embedding models. In particular, ConvKB obtains the best mean rank and the highest Hits@10 on WN18RR, and produces the highest mean reciprocal rank and highest Hits@10 on FB15k-237.

## 2 Proposed ConvKB model

A knowledge base \(\mathcal{G}\) is a collection of valid factual triples in the form of _(head entity, relation, tail entity)_ denoted as \((h,r,t)\) such that \(h,t\in\mathcal{E}\) and \(r\in\mathcal{R}\) where \(\mathcal{E}\) is a set of entities and \(\mathcal{R}\) is a set of relations. Embedding models aim to define a _score function_\(f\) giving an implausibility score for each triple \((h,r,t)\) such that valid triples receive lower scores than invalid triples. Table 1 presents score functions in previous SOTA models.

We denote the dimensionality of embeddings by \(k\) such that each embedding triple (\({\mathbf{v}_{h}}\), \({\mathbf{v}_{r}}\), \({\mathbf{v}_{t}}\)) are

\begin{table}
\begin{tabular}{l|p{142.3pt}} Model & The score function \(f(h,r,t)\) \\ \hline \hline TransE & \(\lVert{\mathbf{v}_{h}+{\mathbf{v}_{r}}\cdot{\mathbf{v}_{t}}}\rVert_{p}\) \\ DISTMULT & \(\langle{\mathbf{v}_{h}},{\mathbf{v}_{r}},{\mathbf{v}_{t}}\rangle\) \\ ComplEx & \(\mathsf{Re}\left(\langle{\mathbf{v}_{h}},{\mathbf{v}_{r}},{\widehat{\mathbf{v}}_{t}}\rangle\right)\) \\ ConvE & \(g\left(\text{vec}\left(g\left(\text{concat}\left({\widehat{\mathbf{v}}_{h}},{ \widehat{\mathbf{v}}_{r}}\right)*\boldsymbol{\Omega}\right)\right){\mathbf{W}}\right) \cdot{\mathbf{v}_{t}}\) \\ \hline \hline ConvKB & \(\textsf{concat}\left(\left[g\left({\mathbf{v}_{h}},{\mathbf{v}_{r}},{\mathbf{v}_{t}}\right]* \boldsymbol{\Omega}\right)\right)\cdot{\bf w}\) \\ \end{tabular}
\end{table}
Table 1: The score functions in previous SOTA models and in our ConvKB model. \(\lVert{\mathbf{v}}\rVert_{p}\) denotes the \(p\)-norm of \({\mathbf{v}}\). \(\langle{\mathbf{v}_{h}},{\mathbf{v}_{r}},{\mathbf{v}_{t}}\rangle=\sum_{i}{\mathbf{v}_{h}}_{{ \mathbf{v}_{r}}}{\mathbf{v}_{t_{i}}}\) denotes a tri-linear dot product. \(g\) denotes a non-linear function. \(*\) denotes a convolution operator. \(\cdot\) denotes a dot product. concat denotes a concatenation operator. \(\widehat{\mathbf{v}}\) denotes a 2D reshaping of \({\mathbf{v}}\). \(\boldsymbol{\Omega}\) denotes a set of filters.

 

viewed as a matrix \({\mathbf{A}}=[{\mathbf{v}_{h},{\mathbf{v}}_{r},{\mathbf{v}}_{t}]}\in\mathbb{R}^{k\times 3}\). And \({\mathbf{A}}_{i,:}\in\mathbb{R}^{1\times 3}\) denotes the \(i\)-th row of \({\mathbf{A}}\). Suppose that we use a filter \({\mathbf{\omega}}\in\mathbb{R}^{1\times 3}\) operated on the convolution layer. \({\mathbf{\omega}}\) is not only aimed to examine the global relationships between same dimensional entries of the embedding triple (\({\mathbf{v}}_{h}\), \({\mathbf{v}}_{r}\), \({\mathbf{v}}_{t}\)), but also to generalize the transitional characteristics in the transition-based models. \({\mathbf{\omega}}\) is repeatedly operated over every row of \({\mathbf{A}}\) to finally generate a feature map \({\mathbf{v}}=[v_{1},v_{2},...,v_{k}]\in\mathbb{R}^{k}\) as:

\[v_{i}=g\left({\mathbf{\omega}}\cdot{\mathbf{A}}_{i,:}+b\right)\]

where \(b\in\mathbb{R}\) is a bias term and \(g\) is some activation function such as ReLU.

Our ConvKB uses different filters \(\in\mathbb{R}^{1\times 3}\) to generate different feature maps. Let \({\bf\Omega}\) and \(\tau\) denote the set of filters and the number of filters, respectively, i.e. \(\tau=|{\bf\Omega}|\), resulting in \(\tau\) feature maps. These \(\tau\) feature maps are concatenated into a single vector \(\in\mathbb{R}^{\tau k\times 1}\) which is then computed with a weight vector \({\mathbf{w}}\in\mathbb{R}^{\tau k\times 1}\) via a dot product to give a score for the triple \((h,r,t)\). Figure 1 illustrates the computation process in ConvKB.

Formally, we define the ConvKB score function \(f\) as follows:

\[f(h,r,t)=\mathsf{concat}\left(g\left([{\mathbf{v}}_{h},{\mathbf{v}}_{r},{\mathbf{v}}_{t}]* {\bf\Omega}\right)\right)\cdot{\mathbf{w}}\]

where \({\bf\Omega}\) and \({\mathbf{w}}\) are shared parameters, independent of \(h\), \(r\) and \(t\); \(*\) denotes a convolution operator; and \(\mathsf{concat}\) denotes a concatenation operator.

If we only use one filter \({\mathbf{\omega}}\) (i.e. using \(\tau=1\)) with a fixed bias term \(b=0\) and the activation function \(g(x)=|x|\) or \(g(x)=x^{2}\), and fix \({\mathbf{\omega}}=[1,1,-1]\) and \({\mathbf{w}}={\textbf{1}}\) during training, ConvKB reduces to the plain TransE model Bordes et al. (2013). So our ConvKB model can be viewed as an extension of TransE to further model global relationships.

We use the Adam optimizer Kingma and Ba (2014) to train ConvKB by minimizing the loss function \(\mathcal{L}\)Trouillon et al. (2016) with \(L_{2}\) regularization on the weight vector \({\mathbf{w}}\) of the model:

\[\begin{split}\mathcal{L}&=\sum_{(h,r,t)\in\{ \mathcal{G}\cup\mathcal{G}^{\prime}\}}\log\big{(}1+\exp\big{(}l_{(h,r,t)}\cdot f \left(h,r,t\right)\big{)}\\ &~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}

 



### Evaluation protocol

In the KB completion or link prediction task (Bordes et al., 2013), the purpose is to predict a missing entity given a relation and another entity, i.e, inferring \(h\) given \((r,t)\) or inferring \(t\) given \((h,r)\). The results are calculated based on ranking the scores produced by the score function \(f\) on test triples.

Following Bordes et al. (2013), for each valid test triple \((h,r,t)\), we replace either \(h\) or \(t\) by each of other entities in \({\cal E}\) to create a set of corrupted triples. We use the "**Filtered**" setting protocol (Bordes et al., 2013), i.e., not taking any corrupted triples that appear in the KB into accounts. We rank the valid test triple and corrupted triples in ascending order of their scores. We employ three common evaluation metrics: mean rank (MR), mean reciprocal rank (MRR), and Hits@10 (i.e., the proportion of the valid test triples ranking in top 10 predictions). Lower MR, higher MRR or higher Hits@10 indicate better performance.

### Training protocol

We use the common Bernoulli trick (Wang et al., 2014; Lin et al., 2015b) to generate the head or tail entities when sampling invalid triples. We also use entity and relation embeddings produced by TransE to _initialize_ entity and relation embeddings in ConvKB. We employ a TransE implementation available at: [https://github.com/datquocnguyen/STransE](https://github.com/datquocnguyen/STransE). We train TransE using a grid search of hyper-parameters: the dimensionality of embeddings \(k\in\{50,100\}\), SGD learning rate \(\in\{1e^{-4},5e^{-4},1e^{-3},5e^{-3}\}\), \(l_{1}\)-norm or \(l_{2}\)-norm, and margin \(\gamma\in\{1,3,5,7\}\). The highest Hits@10 scores on the validation set are when using \(l_{1}\)-norm, learning rate at \(5e^{-4}\), \(\gamma\) = 5 and \(k\) = 50 for WN18RR, and using \(l_{1}\)-norm, learning rate at \(5e^{-4}\), \(\gamma\) = 1 and k = 100 for FB15k-237.

To learn our model parameters including entity and relation embeddings, filters \(\mathbf{\omega}\) and the weight vector \(\mathbf{w}\), we use Adam (Kingma and Ba, 2014) and select its initial learning rate \(\in\{5e^{-6},1e^{-5},5e^{-5},1e^{-4},5e^{-4}\}\). We use ReLU as the activation function \(g\). We fix the batch size at 256 and set the \(L_{2}\)-regularizer \(\lambda\) at 0.001 in our objective function. The filters \(\mathbf{\omega}\) are initialized by a truncated normal distribution or by \([0.1,0.1,-0.1]\). We select the number of filters \(\tau\in\{50,100,200,400,500\}\). We run ConvKB up to 200 epochs and use outputs from the last epoch for evaluation. The highest Hits@10 scores on the validation set are obtained when using \(k\) = 50, \(\tau=500\), the truncated normal distribution for filter initialization, and the initial learning rate at \(1e^{-4}\) on WN18RR; and k = 100, \(\tau=50\), \([0.1,0.1,-0.1]\) for filter initialization, and the initial learning rate at \(5e^{-6}\) on FB15k-237.

### Main experimental results

Table 3 compares the experimental results of our ConvKB model with previous published results, using the same experimental setup. Table 3 shows that ConvKB obtains the best MR and highest Hits@10 scores on WN18RR and also the highest

\begin{table}
\begin{tabular}{p{99.6pt}|c c c|c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c|}{**WN18RR**} & \multicolumn{3}{c}{**FB15k-237**} \\ \cline{2-7}  & MR & MRR & H@10 & MR & MRR & H@10 \\ \hline IRN (Shen et al., 2017) & – & – & – & 211 & – & 46.4 \\ KBGAN (Cai and Wang, 2018) & – & 0.213 & 48.1 & – & 0.278 & 45.8 \\ DISTMULT (Yang et al., 2015) [\(\star\)] & 5110 & 0.43 & 49 & 254 & 0.241 & 41.9 \\ ComplEx (Trouillon et al., 2016) [\(\star\)] & 5261 & 0.44 & 51 & 339 & 0.247 & 42.8 \\ ConvE (Dettmers et al., 2018) & 5277 & **0.46** & 48 & 246 & 0.316 & 49.1 \\ TransE (Bordes et al., 2013) (our results) & 3384 & 0.226 & 50.1 & 347 & 0.294 & 46.5 \\ \hline Our ConvKB model & **2554** & 0.248 & **52.5** & 257 & **0.396** & **51.7** \\ \hline KB\({}_{LRN}\) (García-Murán and Niepert, 2017) & – & – & – & **209** & 0.309 & 49.3 \\ R-GCN+ (Schlichtkrull et al., 2017) & – & – & – & – & 0.249 & 41.7 \\ Neural LP (Yang et al., 2017) & – & – & – & – & 0.240 & 36.2 \\ Node+LinkFeat (Toutanova and Chen, 2015) & – & – & – & – & 0.293 & 46.2 \\ \hline \end{tabular}
\end{table}
Table 3: Experimental results on WN18RR and FB15k-237 test sets. MRR and H@10 denote the mean reciprocal rank and Hits@10 (in %), respectively. [\(\star\)]: Results are taken from Dettlmers et al. (2018) where Hits@10 and MRR are rounded to 2 decimal places on WN18RR. The last 4 rows report results of models that exploit information about relation paths (KB\({}_{LRN}\), R-GCN+ and Neural LP) or textual mentions derived from a large external corpus (Node+LinkFeat). The best score is in **bold**, while the second best score is in underline.



MRR and Hits@10 scores on FB15k-237.

ConvKB does better than the closely related model TransE on both experimental datasets, especially on FB15k-237 where ConvKB gains significant improvements of \(347-257=90\) in MR (which is about 26% relative improvement) and \(0.396-0.294=0.102\) in MRR (which is 34+% relative improvement), and also obtains \(51.7-46.5=5.2\)% absolute improvement in Hits@10. Previous work shows that TransE obtains very competitive results Lin et al. (2015a); Nickel et al. (2016b); Trouillon et al. (2016); Nguyen et al. (2016). However, when comparing the CNN-based embedding model ConvE with other models, Dettmers et al. (2018) did not experiment with TransE. We reconfirm previous findings that TransE in fact is a strong baseline model, e.g., TransE obtains better MR and Hits@10 than ConvE on WN18RR.

ConvKB obtains better scores than ConvE on both datasets (except MRR on WN18RR and MR on FB15k-237), thus showing the usefulness of taking transitional characteristics into accounts. In particular, on FB15k-237, ConvKB achieves improvements of \(0.394-0.316=0.078\) in MRR (which is about 25% relative improvement) and \(51.7-49.1=2.6\)% in Hits@10, while both ConvKB and ConvE produce similar MR scores. ConvKB also obtains 25% relatively higher MRR score than the relation path-based model KB\({}_{LRN}\) on FB15k-237. In addition, ConvKB gives better Hits@10 than KB\({}_{LRN}\), however, KB\({}_{LRN}\) gives better MR than ConvKB. We plan to extend ConvKB with relation path information to obtain better link prediction performance in future work.

## 4 Conclusion

In this paper, we propose a novel embedding model ConvKB for the knowledge base completion task. ConvKB applies the convolutional neural network to explore the global relationships among same dimensional entries of the entity and relation embeddings, so that ConvKB generalizes the transitional characteristics in the transition-based embedding models. Experimental results show that our model ConvKB outperforms other state-of-the-art models on two benchmark datasets WN18RR and FB15k-237. Our code is available at: [https://github.com/daiquccnguyen/ConvKB](https://github.com/daiquccnguyen/ConvKB).

We also plan to extend ConvKB for a new application where we could formulate data in the form of triples. For example, inspired from the work by Vu et al. (2017) for search personalization, we can also apply ConvKB to model _user_-oriented relationships between submitted _queries_ and _documents_ returned by search engines, i.e. modeling triple representations (query, user, document).

## Acknowledgments

This research was partially supported by the Australian Research Council (ARC) Discovery Grant Project DP160103934.

## References

* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_, pages 1247-1250.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multirelational Data. In _Advances in Neural Information Processing Systems 26_, pages 2787-2795.
* Bordes et al. (2011) Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning Structured Embeddings of Knowledge Bases. In _Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence_, pages 301-306.
* Cai and Wang (2018) Liwei Cai and William Yang Wang. 2018. KBGAN: Adversarial Learning for Knowledge Graph Embeddings. In _Proceedings of The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, page to appear.
* Collebert et al. (2011) Ronan Collebert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stetenorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In _Proceedings of the 32nd AAAI Conference on Artificial Intelligence_, page to appear.
* Garcia-Duran and Niepert (2017) Alberto Garcia-Duran and Mathias Niepert. 2017. KBLRN: End-to-end learning of knowledge base representations with latent, relational, and numerical features. _arXiv preprint abs/1709.04676_.
* Guu et al. (2015) Kelvin Guu, John Miller, and Percy Liang. 2015. Traversing Knowledge Graphs in Vector Space. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 318-327 

Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 221-231.
* Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 687-696.
* Ji et al. (2016) Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2016. Knowledge Graph Completion with Adaptive Sparse Transfer Matrix. In _Proceedings of the Thirtieth Conference on Artificial Intelligence_, pages 985-991.
* Kasneci et al. (2008) Gjergji Kasneci, Fabian M Suchanek, Georgiana Ifrim, Maya Ramanath, and Gerhard Weikum. 2008. Naga: Searching and ranking knowledge. In _Proceedings of the 24th IEEE International Conference on Data Engineering_, pages 953-962.
* Kim (2014) Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1746-1751.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* LeCun et al. (1998) Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. _Proceedings of the IEEE_, 86:2278-2324.
* Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Soren Auer, et al. 2015. DBpedia-a large-scale, multilingual knowledge base extracted from Wikipedia. _Semantic Web_, 6:167-195.
* Lin et al. (2015a) Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015a. Modeling Relation Paths for Representation Learning of Knowledge Bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 705-714.
* Lin et al. (2015b) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Learning_, pages 2181-2187.
* Luo et al. (2015) Yuanfei Luo, Quan Wang, Bin Wang, and Li Guo. 2015. Context-Dependent Knowledge Graph Embedding. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1656-1661.
* Nguyen (2017) Dat Quoc Nguyen. 2017. An overview of embedding models of entities and relationships for knowledge base completion. _arXiv preprint_, arXiv:1703.08098.
* Nguyen et al. (2016a) Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. 2016a. Neighborhood Mixture Model for Knowledge Base Completion. In _Proceedings of The 2nd SIGKLL Conference on Computational Natural Language Learning_, pages 40-50.
* Nguyen et al. (2016b) Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. 2016b. STransE: a novel embedding model of entities and relationships in knowledge bases. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 460-466.
* Nickel et al. (2016a) Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016a. A Review of Relational Machine Learning for Knowledge Graphs. _Proceedings of the IEEE_, 104(1):11-33.
* Nickel et al. (2016b) Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016b. Holographic Embeddings of Knowledge Graphs. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, pages 1955-1961.
* Schlichtkrull et al. (2017) Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Riansne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. _arXiv preprint arXiv:1703.06103_.
* Schulmacher and Ponzetto (2014) Michael Schulmacher and Simone Paolo Ponzetto. 2014. Knowledge-based graph document modeling. In _Proceedings of the 7th ACM International Conference on Web Search and Data Mining_, pages 543-552.
* Shen et al. (2017) Yelong Shen, Po-Sen Huang, Ming-Wei Chang, and Jianfeng Gao. 2017. Traversing knowledge graph in vector space without symbolic space guidance. _arXiv preprint arXiv:1611.04642v4_.
* Socher et al. (2013) Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In _Advances in Neural Information Processing Systems 26_, pages 926-934.
* Suchanek et al. (2007)Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In _Proceedings of the 16th International Conference on World Wide Web_, pages 697-706.
* Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015. Observed Versus Latent Features for Knowledge Base and Text Inference. In _Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality_, pages 57-66.

 

Kristina Toutanova, Xi Victoria Lin, Wen tau Yih, Hoifung Poon, and Chris Quirk. 2016. Compositional Learning of Embeddings for Relation Paths in Knowledge Bases and Text. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1434-1444.
* Trouillon et al. (2016) Theo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In _Proceedings of the 33nd International Conference on Machine Learning_, pages 2071-2080.
* Vu et al. (2017) Thanh Vu, Dat Quoc Nguyen, Mark Johnson, Dawei Song, and Alistair Willis. 2017. Search Personalization with Embeddings. In _Proceedings of the 39th European Conference on Information Retrieval_, pages 598-604.
* Wang et al. (2014) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. In _Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence_, pages 1112-1119.
* West et al. (2014) Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge Base Completion via Search-based Question Answering. In _Proceedings of the 23rd International Conference on World Wide Web_, pages 515-526.
* Xiong et al. (2017) Chenyan Xiong, Russell Power, and Jamie Callan. 2017. Explicit semantic ranking for academic search via knowledge graph embedding. In _Proceedings of the 26th International Conference on World Wide Web_, pages 1271-1279.
* Yang and Mitchell (2017) Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in lstms for improving machine reading. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1436-1446.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In _Proceedings of the International Conference on Learning Representations_.
* Yang et al. (2017) Fan Yang, Zhilin Yang, and William W Cohen. 2017. Differentiable Learning of Logical Rules for Knowledge Base Reasoning. pages 2316-2325.
* Zhang et al. (2016) Yuanzhe Zhang, Kang Liu, Shizhu He, Guoliang Ji, Zhanyi Liu, Hua Wu, and Jun Zhao. 2016. Question answering over knowledge base with neural attention combining global knowledge information. _arXiv preprint arXiv:1606.00979_