A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network	In this paper, we propose a novel embedding model, named ConvKB, for knowledge base completion. Our model ConvKB advances state-of-the-art models by employing a convolutional neural network, so that it can capture global relationships and transitional characteristics between entities and relations in knowledge bases. In ConvKB, each triple (head entity, relation, tail entity) is represented as a 3-column matrix where each column vector represents a triple element. This 3-column matrix is then fed to a convolution layer where multiple filters are operated on the matrix to generate different feature maps. These feature maps are then concatenated into a single feature vector representing the input triple. The feature vector is multiplied with a weight vector via a dot product to return a score. This score is then used to predict whether the triple is valid or not. Experiments show that ConvKB achieves better link prediction performance than previous state-of-the-art embedding models on two benchmark datasets WN18RR and FB15k-237.

Furthermore, DISTMULT  and ComplEx  use a tri-linear dot product to compute the score for each triple. Recent research has shown that using relation paths between entities in the KBs could help to get contextual information for improving KB completion performance  ,  ,  ,  ,  . See other embedding models for KB completion in  . Recently, convolutional neural networks (CNNs), originally designed for computer vision  , have significantly received research attention in natural language processing  ,  . CNN learns non-linear features to capture complex relationships with a remarkably less number of parameters compared to fully connected neural networks. Inspired from the success in computer vision,  proposed ConvE—the first model applying CNN for the KB completion task. In ConvE, only $v_h$ and $v_r$ are reshaped and then concatenated into an input matrix which is fed to the convolution layer. Different filters of the same $3\times 3$ shape are operated over the input matrix to output feature map tensors. These feature map tensors are then vectorized and mapped into a vector via a linear transformation.

For each triple (h, r, t), the corresponding triple of $k$ -dimensional embeddings ( $v_h$ , $v_h$0 , $v_h$1 ) is represented as a $v_h$2 input matrix. This input matrix is fed to the convolution layer where different filters of the same $v_h$3 shape are used to extract the global relationships among same dimensional entries of the embedding triple. That is, these filters are repeatedly operated over every row of the input matrix to produce different feature maps. The feature maps are concatenated into a single feature vector which is then computed with a weight vector via a dot product to produce a score for the triple (h, r, t). This score is used to infer whether the triple (h, r, t) is valid or not. Our contributions in this paper are as follows:

Suppose that we use a filter $\omega  \in \mathbb {R}^{1\times 3}$ operated on the convolution layer. $\omega $ is not only aimed to examine the global relationships between same dimensional entries of the embedding triple ( $v_h$0 , $v_h$1 , $v_h$2 ), but also to generalize the transitional characteristics in the transition-based models. $v_h$3 is repeatedly operated over every row of $v_h$4 to finally generate a feature map $v_h$5 as:  $$v_i = g\left(\omega  \cdot {A_{i,:}} + b\right) \nonumber $$   (Eq. 4)  where $b \in \mathbb {R}$ is a bias term and $g$ is some activation function such as ReLU. Our ConvKB uses different filters $\in \mathbb {R}^{1\times 3}$ to generate different feature maps. Let ${\Omega }$ and $\tau $ denote the set of filters and the number of filters, respectively, i.e. $\tau = |{\Omega }|$ , resulting in $\tau $ feature maps.

Main experimental results	Table 3 compares the experimental results of our ConvKB model with previous published results, using the same experimental setup. Table 3 shows that ConvKB obtains the best MR and highest Hits@10 scores on WN18RR and also the highest MRR and Hits@10 scores on FB15k-237. ConvKB does better than the closely related model TransE on both experimental datasets, especially on FB15k-237 where ConvKB gains significant improvements of $347-257 = 90$ in MR (which is about 26% relative improvement) and $0.396 - 0.294 = 0.102$ in MRR (which is 34+% relative improvement), and also obtains $51.7 - 46.5 = 5.2$ % absolute improvement in Hits@10. Previous work shows that TransE obtains very competitive results  ,  ,  ,  . However, when comparing the CNN-based embedding model ConvE with other models,  did not experiment with TransE. We reconfirm previous findings that TransE in fact is a strong baseline model, e.g., TransE obtains better MR and Hits@10 than ConvE on WN18RR.

These $\tau $ feature maps are concatenated into a single vector $\in \mathbb {R}^{\tau k\times 1}$ which is then computed with a weight vector ${w} \in \mathbb {R}^{\tau k\times 1}$ via a dot product to give a score for the triple $(h, r, t)$ . Figure 1 illustrates the computation process in ConvKB. Formally, we define the ConvKB score function $f$ as follows:  $$f(h,r,t) = \mathsf {concat}\left(g\left([v_h,v_r,v_t]\ast {\Omega }\right)\right)\cdot {w} \nonumber $$   (Eq. 6)  where ${\Omega }$ and ${w}$ are shared parameters, independent of $h$ , $r$ and $t$ ; $\ast $ denotes a convolution operator; and $\mathsf {concat}$ denotes a concatenation operator. If we only use one filter $\omega $ (i.e.

Conclusion	In this paper, we propose a novel embedding model ConvKB for the knowledge base completion task. ConvKB applies the convolutional neural network to explore the global relationships among same dimensional entries of the entity and relation embeddings, so that ConvKB generalizes the transitional characteristics in the transition-based embedding models. Experimental results show that our model ConvKB outperforms other state-of-the-art models on two benchmark datasets WN18RR and FB15k-237. Our code is available at: https://github.com/daiquocnguyen/ConvKB. We also plan to extend ConvKB for a new application where we could formulate data in the form of triples. For example, inspired from the work by  for search personalization, we can also apply ConvKB to model user-oriented relationships between submitted queries and documents returned by search engines, i.e. modeling triple representations (query, user, document).

Acknowledgments	This research was partially supported by the Australian Research Council (ARC) Discovery Grant Project DP160103934.

TransE employs a transitional characteristic to model relationships between entities, in which it assumes that if (h, r, t) is a valid fact, the embedding of head entity $h$ plus the embedding of relation $r$ should be close to the embedding of tail entity $t$ , i.e. $v_h$ + $v_r$ $\approx $ $v_t$ (here, $v_h$ , $v_r$ and $h$0 are embeddings of $h$1 , $h$2 and $h$3 respectively). That is, a TransE score $h$4 of the valid triple (h, r, t) should be close to 0 and smaller than a score $h$5 of an invalid triple (h', r', t'). The transitional characteristic in TransE also implies the global relationships among same dimensional entries of $h$6 , $h$7 and $h$8 . Other transition-based models extend TransE to additionally use projection vectors or matrices to translate head and tail embeddings into the relation vector space, such as: TransH  , TransR  , TransD  , STransE  and TranSparse  .

Then this vector is computed with $v_t$ via a dot product to return a score for (h, r, t). See a formal definition of the ConvE score function in Table 1 . It is worth noting that ConvE focuses on the local relationships among different dimensional entries in each of $v_h$ or $v_r$ , i.e., ConvE does not observe the global relationships among same dimensional entries of an embedding triple ( $v_h$ , $v_r$ , $v_t$ ), so that ConvE ignores the transitional characteristic in transition-based models, which is one of the most useful intuitions for the task. In this paper, we present ConvKB—an embedding model which proposes a novel use of CNN for the KB completion task. In ConvKB, each entity or relation is associated with an unique $k$ -dimensional embedding. Let $v_h$ , $v_r$ and $v_t$ denote $k$ -dimensional embeddings of $h$ , $r$ and $t$ , respectively.

Proposed ConvKB model	A knowledge base $\mathcal {G}$ is a collection of valid factual triples in the form of (head entity, relation, tail entity) denoted as $(h, r, t)$ such that $h, t \in \mathcal {E}$ and $r \in \mathcal {R}$ where $\mathcal {E}$ is a set of entities and $\mathcal {R}$ is a set of relations. Embedding models aim to define a score function $f$ giving an implausibility score for each triple $(h, r, t)$ such that valid triples receive lower scores than invalid triples. Table 1 presents score functions in previous SOTA models. We denote the dimensionality of embeddings by $k$ such that each embedding triple ( $v_h$ , $v_r$ , $v_t$ ) are viewed as a matrix $A = [v_h,v_r,v_t] \in \mathbb {R}^{k\times 3}$ . And $A_{i,:} \in \mathbb {R}^{1\times 3}$ denotes the $i$ -th row of $A$ .

using $\tau =1$ ) with a fixed bias term $b=0$ and the activation function $g(x)=|x|$ or $g(x)=x^2$ , and fix $\omega  = [1, 1, -1]$ and ${w} = \textbf {1}$ during training, ConvKB reduces to the plain TransE model  . So our ConvKB model can be viewed as an extension of TransE to further model global relationships. We use the Adam optimizer  to train ConvKB by minimizing the loss function $\mathcal {L}$  with $L_2$ regularization on the weight vector ${w}$ of the model:  $$\text{in which, } l_{(h,r,t)} = \left\lbrace 
\begin{array}{l}
1\;\text{for } (h,r,t)\in \mathcal {G}\\
-1\;\text{for } (h,r,t)\in \mathcal {G}^{\prime }
\end{array} \right.$$   (Eq. )

$
\mathcal {L} & = \sum _{\begin{array}{c}(h,r,t) \in \lbrace \mathcal {G} \cup \mathcal {G}^{\prime }\rbrace \end{array}} \log \left(1 + \exp \left(l_{(h,r,t)} \cdot f\left(h,r,t\right)\right)\right) \nonumber \\
& \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ + \frac{\lambda }{2}\Vert {w}\Vert ^2_2 \nonumber 
$  here $\mathcal {G}^{\prime }$ is a collection of invalid triples generated by corrupting valid triples in $\mathcal {G}$ .

Datasets	We evaluate ConvKB on two benchmark datasets: WN18RR  and FB15k-237  . WN18RR and FB15k-237 are correspondingly subsets of two common datasets WN18 and FB15k  . As noted by  , WN18 and FB15k are easy because they contain many reversible relations. So knowing relations are reversible allows us to easily predict the majority of test triples, e.g. state-of-the-art results on both WN18 and FB15k are obtained by using a simple reversal rule as shown in  . Therefore, WN18RR and FB15k-237 are created to not suffer from this reversible relation problem in WN18 and FB15k, for which the knowledge base completion task is more realistic. Table 2 presents the statistics of WN18RR and FB15k-237.

Evaluation protocol	In the KB completion or link prediction task  , the purpose is to predict a missing entity given a relation and another entity, i.e, inferring $h$ given $(r, t)$ or inferring $t$ given $(h, r)$ . The results are calculated based on ranking the scores produced by the score function $f$ on test triples. Following  , for each valid test triple $(h, r, t)$ , we replace either $h$ or $t$ by each of other entities in $\mathcal {E}$ to create a set of corrupted triples. We use the “Filtered” setting protocol  , i.e., not taking any corrupted triples that appear in the KB into accounts. We rank the valid test triple and corrupted triples in ascending order of their scores. We employ three common evaluation metrics: mean rank (MR), mean reciprocal rank (MRR), and Hits@10 (i.e., the proportion of the valid test triples ranking in top 10 predictions). Lower MR, higher MRR or higher Hits@10 indicate better performance.

Training protocol	We use the common Bernoulli trick  ,  to generate the head or tail entities when sampling invalid triples. We also use entity and relation embeddings produced by TransE to initialize entity and relation embeddings in ConvKB. We employ a TransE implementation available at: https://github.com/datquocnguyen/STransE. We train TransE for 3,000 epochs, using a grid search of hyper-parameters: the dimensionality of embeddings $k \in \lbrace 50, 100\rbrace $ , SGD learning rate $\in \lbrace 1e^{-4}, 5e^{-4}, 1e^{-3}, 5e^{-3}\rbrace $ , $\mathit {l}_1$ -norm or $\mathit {l}_2$ -norm, and margin $\gamma \in \lbrace 1, 3, 5, 7\rbrace $ .

The highest Hits@10 scores on the validation set are when using $\mathit {l}_1$ -norm, learning rate at $5e^{-4}$ , $\gamma $ = 5 and $k$ = 50 for WN18RR, and using $\mathit {l}_1$ -norm, learning rate at $\in \lbrace 1e^{-4}, 5e^{-4}, 1e^{-3}, 5e^{-3}\rbrace $0 , $\in \lbrace 1e^{-4}, 5e^{-4}, 1e^{-3}, 5e^{-3}\rbrace $1 = 1 and k = 100 for FB15k-237. To learn our model parameters including entity and relation embeddings, filters $\omega $ and the weight vector ${w}$ , we use Adam  and select its initial learning rate $\in \lbrace 5e^{-6}, 1e^{-5}, 5e^{-5}, 1e^{-4}, 5e^{-4}\rbrace $ . We use ReLU as the activation function $g$ .

We fix the batch size at 256 and set the $L_2$ -regularizer $\lambda $ at 0.001 in our objective function. The filters $\omega $ are initialized by a truncated normal distribution or by $[0.1, 0.1, -0.1]$ . We select the number of filters $\tau \in \lbrace 50, 100, 200, 400, 500\rbrace $ . We run ConvKB up to 200 epochs and use outputs from the last epoch for evaluation. The highest Hits@10 scores on the validation set are obtained when using $k$ = 50, ${w}$0 , the truncated normal distribution for filter initialization, and the initial learning rate at ${w}$1 on WN18RR; and k = 100, ${w}$2 , ${w}$3 for filter initialization, and the initial learning rate at ${w}$4 on FB15k-237.

ConvKB obtains better scores than ConvE on both datasets (except MRR on WN18RR and MR on FB15k-237), thus showing the usefulness of taking transitional characteristics into accounts. In particular, on FB15k-237, ConvKB achieves improvements of $0.394-0.316 = 0.078$ in MRR (which is about 25% relative improvement) and $51.7 - 49.1 = 2.6$ % in Hits@10, while both ConvKB and ConvE produce similar MR scores. ConvKB also obtains 25% relatively higher MRR score than the relation path-based model KB $_{LRN}$ on FB15k-237. In addition, ConvKB gives better Hits@10 than KB $_{LRN}$ , however, KB $_{LRN}$ gives better MR than ConvKB. We plan to extend ConvKB with relation path information to obtain better link prediction performance in future work.

Introduction	Large-scale knowledge bases (KBs), such as YAGO  , Freebase  and DBpedia  , are usually databases of triples representing the relationships between entities in the form of fact (head entity, relation, tail entity) denoted as (h, r, t), e.g., (Melbourne, cityOf, Australia). These KBs are useful resources in many applications such as semantic searching and ranking  ,  ,  , question answering  ,  and machine reading  . However, the KBs are still incomplete, i.e., missing a lot of valid triples  ,  . Therefore, much research work has been devoted towards knowledge base completion or link prediction to predict whether a triple (h, r, t) is valid or not  . Many embedding models have proposed to learn vector or matrix representations for entities and relations, obtaining state-of-the-art (SOTA) link prediction results  . In these embedding models, valid triples obtain lower implausibility scores than invalid triples. Let us take the well-known embedding model TransE  as an example. In TransE, entities and relations are represented by $k$ -dimensional vector embeddings.

Question: Did the authors try stacking multiple convolutional layers?
Answer: No