Venue: NAACL 2019

# A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization

Dai Quoc Nguyen\({}^{1}\), Thanh Vu\({}^{2}\), Tu Dinh Nguyen\({}^{1}\), Dat Quoc Nguyen\({}^{3}\), Dinh Phung\({}^{1}\)

\({}^{1}\)Monash University, Australia; \({}^{3}\)The University of Melbourne, Australia

\({}^{2}\)The Australian e-Health Research Centre, CSIRO, Australia

\({}^{1}\){dai.nguyen,tu.dinh.nguyen,dinh.phung}@monash.edu

\({}^{2}\)thanh.vu@csiro.au; \({}^{3}\)dqnguyen@unimelb.edu.au

###### Abstract

In this paper, we introduce an embedding model, named CapsE, exploring a capsule network to model relationship triples _(subject, relation, object)_. Our CapsE represents each triple as a 3-column matrix where each column vector represents the embedding of an element in the triple. This 3-column matrix is then fed to a convolution layer where multiple filters are operated to generate different feature maps. These feature maps are reconstructed into corresponding capsules which are then routed to another capsule to produce a continuous vector. The length of this vector is used to measure the plausibility score of the triple. Our proposed CapsE obtains better performance than previous state-of-the-art embedding models for knowledge graph completion on two benchmark datasets WN18RR and FB15k-237, and outperforms strong search personalization baselines on SEARCH17.

## 1 Introduction

Knowledge graphs (KGs) containing relationship triples _(subject, relation, object)_, denoted as _(s, r, o)_, are the useful resources for many NLP and especially information retrieval applications such as semantic search and question answering Wang et al. (2017). However, large knowledge graphs, even containing billions of triples, are still incomplete, i.e., missing a lot of valid triples West et al. (2014). Therefore, much research efforts have focused on the knowledge graph completion task which aims to predict missing triples in KGs, i.e., predicting whether a triple not in KGs is likely to be valid or not Bordes et al. (2011, 2013); Socher et al. (2013). To this end, many embedding models have been proposed to learn vector representations for entities (i.e., _subject_/head entity and _object_/tail entity) and relations in KGs, and obtained state-of-the-art results as summarized by Nickel et al. (2016a) and Nguyen (2017). These embedding models score triples _(s, r, o)_, such that valid triples have higher plausibility scores than invalid ones Bordes et al. (2011, 2013); Socher et al. (2013). For example, in the context of KGs, the score for _(Melbourne, cityOf, Australia)_ is higher than the score for _(Melbourne, cityOf, United Kingdom)_.

Triple modeling is applied not only to the KG completion, but also for other tasks which can be formulated as a triple-based prediction problem. An example is in search personalization, one would aim to tailor search results to each specific user based on the user's personal interests and preferences Tevan et al. (2005, 2009); Bennett et al. (2012); Harvey et al. (2013); Vu et al. (2015, 2017). Here the triples can be formulated as _(submitted query, user profile, returned document)_ and used to re-rank documents returned to a user given an input query, by employing an existing KG embedding method such as TransE Bordes et al. (2013), as proposed by Vu et al. (2017). Previous studies have shown the effectiveness of modeling triple for either KG completion or search personalization. However, there has been no single study investigating the performance on both tasks.

Conventional embedding models, such as TransE Bordes et al. (2013), DISTMULT Yang et al. (2015) and ComplEx Trouillon et al. (2016), use addition, subtraction or simple multiplication operators, thus only capture the linear relationships between entities. Recent research has raised interest in applying deep neural networks to triple-based prediction problems. For example, Nguyen et al. (2018) proposed ConvKB--a convolutional neural network (CNN)-based model for KG completion and achieved state-of-the-art results. Most of KG embedding models are constructed to modeling entries at the same dimension of the given triple, where presumably each dimension captures some relation-specific attribute of entities. To the

best of our knowledge, however, none of the existing models has a "deep" architecture for modeling the entries in a triple at the same dimension.

Sabour et al. (2017) introduced capsule networks (CapsNet) that employ capsules (i.e., _each capsule is a group of neurons_) to capture entities in images and then uses a routing process to specify connections from capsules in a layer to those in the next layer. Hence CapsNet could encode the intrinsic spatial relationship between a part and a whole constituting viewpoint invariant knowledge that automatically generalizes to novel viewpoints. Each capsule accounts for capturing variations of an object or object part in the image, which can be efficiently visualized. Our high-level hypothesis is that embedding entries at the same dimension of the triple also have these variations, although it is not straightforward to be visually examined.

To that end, we introduce CapsE to explore a novel application of CapsNet on triple-based data for two problems: KG completion and search personalization. Different from the traditional modeling design of CapsNet where capsules are constructed by splitting feature maps, we use capsules to model the entries at the same dimension in the entity and relation embeddings. In our CapsE, \({\mathbf{v}}_{s}\), \({\mathbf{v}}_{r}\) and \({\mathbf{v}}_{o}\) are unique \(k\)-dimensional embeddings of \(s\), \(r\) and \(o\), respectively. The embedding triple [\({\mathbf{v}}_{s}\), \({\mathbf{v}}_{r}\), \({\mathbf{v}}_{o}\)] of _(s, r, o)_ is fed to the convolution layer where multiple filters of the same \(1\times 3\) shape are repeatedly operated over every row of the matrix to produce \(k\)-dimensional feature maps. Entries at the same dimension from all feature maps are then encapsulated into a capsule. Thus, _each capsule can encode many characteristics in the embedding triple to represent the entries at the corresponding dimension_. These capsules are then routed to another capsule which outputs a continuous vector whose length is used as a score for the triple. Finally, this score is used to predict whether the triple _(s, r, o)_ is valid or not.

In summary, our main contributions from this paper are as follows:

\(\bullet\) We propose an embedding model CapsE using the capsule network Sabour et al. (2017) for modeling relationship triples. To our best of knowledge, our work is the first consideration of exploring the capsule network to knowledge graph completion and search personalization.

\(\bullet\) We evaluate our CapsE for knowledge graph completion on two benchmark datasets WN18RR Dettmers et al. (2018) and FB15k-237 Toutanova and Chen (2015). CapsE obtains the best mean rank on WN18RR and the highest mean reciprocal rank and highest Hits@10 on FB15k-237.

\(\bullet\) We restate the prospective strategy of expanding the triple embedding models to improve the ranking quality of the search personalization systems. We adapt our model to search personalization and evaluate on SEARCH17 Vu et al. (2017) - a dataset of the web search query logs. Experimental results show that our CapsE achieves the new state-of-the-art results with significant improvements over strong baselines.

## 2 The proposed CapsE

Let \(\mathcal{G}\) be a collection of valid factual triples in the form of _(subject, relation, object)_ denoted as _(s, r, o)_. Embedding models aim to define a _score function_ giving a score for each triple, such that valid triples receive higher scores than invalid triples.

We denote \({\mathbf{v}}_{s}\), \({\mathbf{v}}_{r}\) and \({\mathbf{v}}_{o}\) as the \(k\)-dimensional embeddings of \(s\), \(r\) and \(o\), respectively. In our proposed CapsE, we follow Nguyen et al. (2018) to view each embedding triple [\({\mathbf{v}}_{s}\), \({\mathbf{v}}_{r}\), \({\mathbf{v}}_{o}\)] as a matrix \({\mathbf{A}}=[{\mathbf{v}}_{s},{\mathbf{v}}_{r},{\mathbf{v}}_{o}]\in\mathbb{R}^{k\times 3}\), and denote \({\mathbf{A}}_{i,:}\in\mathbb{R}^{1\times 3}\) as the \(i\)-th row of \({\mathbf{A}}\). We use a filter \({\mathbf{\omega}}\in\mathbb{R}^{1\times 3}\) operated on the convolution layer. This filter \({\mathbf{\omega}}\) is repeatedly operated over every row of \({\mathbf{A}}\) to generate a feature map \({\bf q}=[{\sf q}_{1},{\sf q}_{2},...,{\sf q}_{k}]\in\mathbb{R}^{k}\), in which \({\sf q}_{i}=g\left({\mathbf{\omega}}\cdot{\mathbf{A}}_{i,:}+b\right)\) where \(\cdot\) denotes a dot product, \(b\in\mathbb{R}\) is a bias term and \(g\) is a non-linear activation function such as ReLU. Our model uses multiple filters \(\in\mathbb{R}^{1\times 3}\) to generate feature maps. We denote \({\mathbf{\Omega}}\) as the set of filters and \({\mathsf{N}}=\mid{\mathbf{\Omega}}\mid\) as the number of filters, thus we have \({\mathsf{N}}\)\(k\)-dimensional feature maps, for which each feature map can capture one single characteristic among entries at the same dimension.

We build our CapsE with two single capsule layers for a simplified architecture. In the first layer, we construct \(k\) capsules, wherein entries at the same dimension from all feature maps are encapsulated into a corresponding capsule. Therefore, each capsule can capture many characteristics among the entries at the corresponding dimension in the embedding triple. These characteristics are generalized into one capsule in the second layer which produces a vector output whose length is used as the score for the triple.

The first capsule layer consists of \(k\) capsules, for which each capsule \(i\in\{1,2,...,k\}\) has a vector 

output \({\bf u}_{i}\in\mathbb{R}^{N\times 1}\). Vector outputs \({\bf u}_{i}\) are multiplied by weight matrices \({\bf W}_{i}\in\mathbb{R}^{d\times N}\) to produce vectors \(\hat{{\bf u}}_{i}\in\mathbb{R}^{d\times 1}\) which are summed to produce a vector input \({\bf s}\in\mathbb{R}^{d\times 1}\) to the capsule in the second layer. The capsule then performs the non-linear squashing function to produce a vector output \({\bf e}\in\mathbb{R}^{d\times 1}\):

where \(\mathsf{squash}\left({\bf s}\right)=\frac{\|{\bf s}\|^{2}}{1+\|{\bf s}\|^{2}} \frac{{\bf s}}{\|{\bf s}\|}\), and \(c_{i}\) are coupling coefficients determined by the routing process as presented in Algorithm 1. Because there is one capsule in the second layer, we make only one difference in the routing process proposed by Sabour et al. (2017), for which we apply the softmax in a direction from all capsules in the previous layer to each of capsules in the next layer.1

Footnote 1: The softmax in the original routing process proposed by Sabour et al. (2017) is applied in another direction from each of capsules in the previous layer to all capsules in the next layer.

``` forallcapsule\(i\in\textit{the first layerdo}\quad b_{i}\gets 0\)foriteration=1,2,...,mdo c softmax\(({\bf b})\) s s 



### Experimental setup

**Datasets:** We use two recent benchmark datasets WN18RR Dettmers et al. (2018) and FB15k-237 Toutanova and Chen (2015). These two datasets are created to avoid reversible relation problems, thus the prediction task becomes more realistic and hence more challenging Toutanova and Chen (2015). Table 1 presents the statistics of WN18RR and FB15k-237.

**Evaluation protocol:** Following Bordes et al. (2013), for each valid test triple \((s,r,o)\), we replace either \(s\) or \(o\) by each of all other entities to create a set of corrupted triples. We use the "Filtered" setting protocol Bordes et al. (2013), i.e., not taking any corrupted triples that appear in the KG into accounts. We rank the valid test triple and corrupted triples in descending order of their scores. We employ evaluation metrics: mean rank (MR), mean reciprocal rank (MRR) and Hits@10 (i.e., the proportion of the valid test triples ranking in top 10 predictions). Lower MR, higher MRR or higher Hits@10 indicate better performance. Final scores on the test set are reported for the model obtaining the highest Hits@10 on the validation set.

**Training protocol:** We use the common Bernoulli strategy Wang et al. (2014); Lin et al. (2015b) when sampling invalid triples. For WN18RR, Pinter and Eisenstein (2018)2 found a strong evidence to support the necessity of a WordNet-related semantic setup, in which they averaged pre-trained word embeddings for word surface forms within the WordNet to create synset embeddings, and then used these synset embeddings to initialize entity embeddings for training their TransE association model. We follow this evidence in using the pre-trained 100-dimensional Glove word embeddings Pennington et al. (2014) to train a TransE model on WN18RR.

Footnote 2: Pinter and Eisenstein (2018) considered WN18RR and evaluated their M3GM model only for 7 relations as they employed the inverse rule model Dettmers et al. (2018) for 4 remaining symmetric relations. Regarding a fair comparison to other models, we use the M3GM implementation released by Pinter and Eisenstein (2018) to re-train and re-evaluate the M3GM model for all 11 relations. We thank Pinter and Eisenstein (2018) for their assistance running their code.

We employ the TransE and ConvKB implementations provided by Nguyen et al. (2016b) and Nguyen et al. (2018). For ConvKB, we use a new process of training up to 100 epochs and monitor the Hits@10 score after every 10 training epochs to choose optimal hyper-parameters with the Adam initial learning rate in \(\{1e^{-5},5e^{-5},1e^{-4}\}\) and the number of filters N in \(\{50,100,200,400\}\). We obtain the highest Hits@10 scores on the validation set when using N= 400 and the initial learning rate \(5e^{-5}\) on WN18RR; and N= 100 and the initial learning rate \(1e^{-5}\) on FB15k-237.

Like in ConvKB, we use the same pre-trained entity and relation embeddings produced by TransE to initialize entity and relation embeddings in our CapsE for both WN18RR and FB15k-237 (\(k=100\)). We set the batch size to 128, the number of neurons within the capsule in the second capsule layer to 10 (\(d=10\)), and the number of iterations in the routing algorithm \(m\) in \(\{1,3,5,7\}\). We run CapsE up to 50 epochs and monitor the Hits@10 score after each 10 training epochs to choose optimal hyper-parameters. The highest Hits@10 scores for our CapsE on the validation set are obtained when using \(m=1\), \(\textsf{N}=400\) and the initial learning rate at \(1e^{-5}\) on WN18RR; and \(m=1\), \(\textsf{N}=50\) and the initial learning rate at \(1e^{-4}\) on FB15k-237.

### Main experimental results

Table 2 compares the experimental results of our CapsE with previous state-of-the-art published results, using the same evaluation protocol. Our CapsE performs better than its closely related CNN-based model ConvKB on both experimental datasets (except Hits@10 on WN18RR and MR on FB15k-237), especially on FB15k-237 where our CapsE gains significant improvements of \(0.523-0.418=0.105\) in MRR (which is about 25.1% relative improvement), and \(59.3\%-53.2\%=6.1\)% absolute improvement in Hits@10. Table 2 also shows that our CapsE obtains the best MR score on WN18RR and the highest MRR and Hits@10 scores on FB15k-237.

Following Bordes et al. (2013), for each relation \(r\) in FB15k-237, we calculate the averaged number \(\eta_{s}\) of head entities per tail entity and the averaged number \(\eta_{o}\) of tail entities per head entity. If \(\eta_{s}<\)1.5 and \(\eta_{o}<\)1.5, \(r\) is categorized one-to-one (1-1). If \(\eta_{s}<\)1.5 and \(\eta_{o}\geq\)1.5, \(r\) is categorized one-to-many (1-M). If \(\eta_{s}\geq\)1.5 and \(\eta_{o}<\)1.5, \(r\) is

\begin{table}
\begin{tabular}{l|c c c c} \hline
**Dataset** & \#E & \#R & \#Triples in train/valid/test \\ \hline WN18RR & 40,943 & 11 & 86,835 & 3,034 & 3,134 \\ FB15k-237 & 14,541 & 237 & 272,115 & 17,535 & 20,466 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the experimental datasets. #E is the number of entities. #R is the number of relations.



categorized many-to-one (M-1). If \(\eta_{s}\geq\)1.5 and \(\eta_{o}\geq\)1.5, \(r\) is categorized many-to-many (M-M). As a result, 17, 26, 81 and 113 relations are labelled 1-1, 1-M, M-1 and M-M, respectively. And 0.9%, 6.3%, 20.5% and 72.3% of the test triples in FB15k-237 contain 1-1, 1-M, M-1 and M-M relations, respectively.

Figure 2 shows the Hits@10 and MRR results for predicting head and tail entities w.r.t each relation category on FB15k-237. CapsE works better than ConvKB in predicting entities on the "side M" of triples (e.g., predicting _head_ entities in M-1 and M-M; and predicting _tail_ entities in 1-M and M-M), while ConvKB performs better than CapsE in predicting entities on the "side 1" of triples (i.e., predicting _head_ entities in 1-1 and 1-M; and predicting _tail_ entities in 1-1 and M-1).

Figure 3 shows the Hits@10 and MRR scores w.r.t each relation on WN18RR. \(also\_see\), \(similar\_to\), \(verb\_group\) and \(derivationally\_related\_form\) are symmetric relations which can be considered as M-M relations. Our CapsE also performs better than ConvKB on these 4 M-M relations. Thus, results

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c|}{**WN18RR**} & \multicolumn{3}{c}{**FB15k-237**} \\ \cline{2-7}  & MR & MRR & H@10 & MR & MRR & H@10 \\ \hline DISTMULT Yang et al. (2015) & 5110 & 0.425 & 49.1 & 254 & 0.241 & 41.9 \\ ComplEx Trouillon et al. (2016) & 5261 & **0.444** & 50.7 & 339 & 0.247 & 42.8 \\ Conv Depttners et al. (2018) & 4187 & 0.433 & 51.5 & **244** & 0.325 & 50.1 \\ KBGAN Cai and Wang (2018) & – & 0.213 & 48.1 & – & 0.278 & 45.8 \\ M3GM Pintera Elisenstein (2018) & 1864 & 0.311 & 53.3 & – & – & — \\ \hline TransE Bordes et al. (2013) & 743\({}^{\star}\) & 0.245\({}^{\star}\) & 56.0\({}^{\star}\) & 347 & 0.294 & 46.5 \\ ConvKB Nguyen et al. (2018) & 763\({}^{\star}\) & 0.253\({}^{\star}\) & **56.7\({}^{\star}\)** & 254\({}^{\star}\) & 0.418\({}^{\star}\) & 53.2\({}^{\star}\) \\ \hline Our **CapsE** & **719** & 0.415 & 56.0 & 303 & **0.523** & **59.3** \\ \hline \end{tabular}
\end{table}
Table 2: Experimental results on the WN18RR and FB15k-237 test sets. Hits@10 (H@10) is reported in %. Results of DISTMULT, ComplEx and ConvE are taken from Dettmers et al. (2018). Results of TransE on FB15k-237 are taken from Nguyen et al. (2018). Our CapsE Hits@1 scores are 33.7% on WN18RR and 48.9% on FB15k-237. Formulas of MRR and Hits@1 show a strong correlation, so using Hits@1 does not really reveal any additional information for this task. The best score is in bold, while the second best score is in underline. \(\star\) denotes _our new results_ for TransE and ConvKB, which are better than those published by Nguyen et al. (2018).

Figure 3: Hits@10 and MRR on the WN18RR test set w.r.t each relation. The right y-axis is the percentage of triples corresponding to relations.

Figure 2: Hits@10 (in %) and MRR on the FB15k-237 test set w.r.t each relation category.

 

shown in Figures 2 and 3 are consistent. These also imply that our CapsE would be a potential candidate for applications which contain many M-M relations such as search personalization.

We see that the length and orientation of each capsule in the first layer can also help to model the important entries in the corresponding dimension, thus CapsE can work well on the "side M" of triples where entities often appear less frequently than others appearing in the "side 1" of triples. Additionally, existing models such as DISTMULT, ComplEx and ConvE can perform well for entities with high frequency, but may not for rare entities with low frequency. These are reasons why our CapsE can be considered as the best one on FB15k-237 and it outperforms most existing models on WN18RR.

**Effects of routing iterations:** We study how the number of routing iterations affect the performance. Table 3 shows the Hits@10 scores on the WN18RR validation set for a comparison w.r.t each number value of the routing iterations and epochs with the number of filters \({\sf N}=50\) and the Adam initial learning rate at \(1e^{-5}\). We see that the best performance for each setup over each 10 epochs is obtained by setting the number \(m\) of routing iterations to 1. This indicates the opposite side for knowledge graphs compared to images. In the image classification task, setting the number \(m\) of iterations in the routing process higher than 1 helps to capture the relative positions of entities in an image (e.g., eyes, nose and mouth) properly. In contrast, this property from images may be only right for the 1-1 relations, but not for the 1-M, M-1 and M-M relations in the KGs because of the high variant of each relation type (e.g., symmetric relations) among different entities.

## 4 Search personalization application

Given a _user_, a submitted _query_ and the _documents_ returned by a search system for that query, our approach is to re-rank the returned documents so that the more relevant documents should be ranked higher. Following Vu et al. (2017), we represent the relationship between the submitted query, the user and the returned document as a _(s, r, o)_-like triple _(query, user, document)_. The triple captures how much interest a user puts on a document given a query. Thus, we can evaluate the effectiveness of our CapsE for the search personalization task.

### Experimental setup

Dataset:We use the SEARCH17 dataset (Vu et al., 2017) of query logs of 106 users collected by a large-scale web search engine. A log entity consists of a user identifier, a query, top-10 ranked documents returned by the search engine and clicked documents along with the user's dwell time. Vu et al. (2017) constructed short-term (session-based) user profiles and used the profiles to personalize the returned results. They then employed the SAT criteria (Fox et al., 2005) to identify whether a returned document is relevant from the query logs as either a clicked document with a dwell time of at least 30 seconds or the last clicked document in a search session (i.e., a SAT click). After that, they assigned a \(relevant\) label to a returned document if it is a SAT click and also assigned \(irrelevant\) labels to the remaining top-10 documents. The rank position of the \(relevant\) labeled documents is used as the ground truth to evaluate the search performance before and after re-ranking.

The dataset was uniformly split into the training, validation and test sets. This split is for the purpose of using historical data in the training set to predict new data in the test set (Vu et al., 2017). The training, validation and test sets consist of 5,658, 1,184 and 1,210 relevant (i.e., valid) triples; and 40,239, 7,882 and 8,540 irrelevant (i.e., invalid) triples, respectively.

Evaluation protocol:Our CapsE is used to re-rank the original list of documents returned by a search engine as follows: (i) We train our model and employ the trained model to calculate the score for each \((s,r,o)\) triple. (ii) We then sort the scores in the descending order to obtain a new ranked list. To evaluate the performance of our proposed model, we use two standard evaluation metrics: mean reciprocal rank (MRR) and Hits@1.3 For each metric, the higher value indi 

cates better ranking performance.

Footnote 10: [https://github.com/journals/csa-research/en](https://github.com/journals/csa-research/en)

We compare CapsE with the following baselines using the same experimental setup: **(1)** SE: The original rank is returned by the search engine. **(2)** CI [35]: This baseline uses a personalized navigation method based on previously clicking returned documents. **(3)** SP [4, 67]: A search personalization method makes use of the session-based user profiles. **(4)** Following [68], we use TransE as a strong baseline model for the search personalization task. Previous work shows that the well-known embedding model TransE, despite its simplicity, obtains very competitive results for the knowledge graph completion [9, 10, 11]. **(5)** The CNN-based model ConvKB is the most closely related model to our CapsE.

**Embedding initialization:** We follow [67] to initialize user profile, query and document embeddings for the baselines TransE and ConvKB, and our CapsE.

We train a LDA topic model [1] with 200 topics only on the _relevant_ documents (i.e., SAT clicks) extracted from the query logs. We then use the trained LDA model to infer the probability distribution over topics for every returned document. We use the topic proportion vector of each document as its document embedding (i.e. \(k=200\)). In particular, the \(z^{th}\) element (\(z=1,2,...,k\)) of the vector embedding for document \(d\) is: \(\mathbf{v}_{d,z}=\mathrm{P}(z~{}|~{}d)\) where \(\mathrm{P}(z~{}|~{}d)\) is the probability of the topic \(z\) given the document \(d\).

We also represent each query by a probability distribution vector over topics. Let \(\mathcal{D}_{q}=\{d_{1},d_{2},...,d_{n}\}\) be the set of top \(n\) ranked documents returned for a query \(q\) (here, \(n=10\)). The \(z^{th}\) element of the vector embedding for query \(q\) is defined as in [67]: \(\mathbf{v}_{q,z}=\sum_{i=1}^{n}\lambda_{i}\mathrm{P}(z~{}|~{}d_{i})\), where \(\lambda_{i}=\frac{\delta^{i-1}}{\sum_{j=1}^{n}\delta^{j-1}}\) is the exponential decay function of \(i\) which is the rank of \(d_{i}\) in \(D_{q}\). And \(\delta\) is the decay hyper-parameter (\(0<\delta<1\)). Following [67], we use \(\delta=0.8\). Note that if we learn query and document embeddings during training, the models will overfit to the data and will not work for new queries and documents. Thus, after the initialization process, we fix (i.e., not updating) query and document embeddings during training for TransE, ConveKB and CapsE.

In addition, as mentioned by Bennett et al. bennett2012learning, the more recently clicked document expresses more about the user current search interest. Hence, we make use of the user clicked documents in the training set with the temporal weighting scheme proposed by [67] to initialize user profile embeddings for the three embedding models.

**Hyper-parameter tuning:** For our CapsE model, we set batch size to 128, and also the number of neurons within the capsule in the second capsule layer to 10 (\(d=10\)). The number of iterations in the routing algorithm is set to 1 (\(m=1\)). For the training model, we use the Adam optimizer with the initial learning rate \(\in\{5e^{-6},1e^{-5},5e^{-5},~{}1e^{-4},~{}5e^{-4}\}\). We also use ReLU as the activation function \(g\). We select the number of filters \(\mathsf{N}\in\{50,100,200,400,500\}\). We run the model up to 200 epochs and perform a grid search to choose optimal hyper-parameters on the validation set. We monitor the MRR score after each training epoch and obtain the highest MRR score on the validation set when using \(\mathsf{N}=400\) and the initial learning rate at \(5e^{-5}\).

We employ the TransE and ConvKB implementations provided by [67] and [67] and then follow their training protocols to tune hyper-parameters for TransE and ConvKB, respectively. We also monitor the MRR score after each training epoch and attain the highest MRR score on the validation set when using margin = 5, \(l_{1}\)-norm and SGD learning rate at \(5e^{-3}\) for TransE; and \(\mathsf{N}=500\) and the Adam initial learning rate at \(5e^{-4}\) for ConvKB.

### Main results

Table 4 presents the experimental results of the baselines and our model. Embedding models TranE, ConvKB and CapsE produce better ranking performances than traditional learning-to-rank search personalization models CI and SP. This indicates a prospective strategy of expanding the triple embedding models to improve the ranking quality of the search personalization systems. In particular, our MRR and Hits@1 scores are higher than those of TransE (with relative improvements of 14.5% and 22% over TransE, respectively). Specifically, our CapsE achieves the highest performances in both MRR and Hits@1 (our improvements over all five baselines are statistically



significant with \(p<0.05\) using the _paired t-test_).

To illustrate our training progress, we plot performances of CapsE on the validation set over epochs in Figure 4. We observe that the performance is improved with the increase in the number of filters since capsules can encode more useful properties for a large embedding size.

## 5 Related work

Other transition-based models extend TransE to additionally use projection vectors or matrices to translate embeddings of \(s\) and \(o\) into the vector space of \(r\), such as: TransH Wang et al. (2014), TransR Lin et al. (2015b), TransD Ji et al. (2015) and STransE Nguyen et al. (2016b). Furthermore, DISTMULT Yang et al. (2015) and ComplEx Trouillon et al. (2016) use a tri-linear dot product to compute the score for each triple. Moreover, ConvKB Nguyen et al. (2018) applies convolutional neural network, in which feature maps are concatenated into a single feature vector which is then computed with a weight vector via a dot product to produce the score for the input triple. ConvKB is the most closely related model to our CapsE. See an overview of embedding models for KG completion in Nguyen (2017).

For search tasks, unlike classical methods, personalized search systems utilize the historical interactions between the user and the search system, such as submitted queries and clicked documents to tailor returned results to the need of that user Teevan et al. (2005, 2009). That historical information can be used to build the _user profile_, which is crucial to an effective search personalization system. Widely used approaches consist of two separated steps: (1) building the user profile from the interactions between the user and the search system; and then (2) learning a ranking function to _re-rank_ the search results using the user profile Bennett et al. (2012); White et al. (2013); Harvey et al. (2013); Vu et al. (2015). The general goal is to re-rank the documents returned by the search system in such a way that the more relevant documents are ranked higher. In this case, apart from the user profile, dozens of other features have been proposed as the input of a learning-to-rank algorithm Bennett et al. (2012); White et al. (2013). Alternatively, Vu et al. (2017) modeled the potential _user_-oriented relationship between the submitted query and the returned document by applying TransE to reward higher scores for more relevant documents (e.g., clicked documents). They achieved better performances than the standard ranker as well as competitive search personalization baselines Teeven et al. (2011); Bennett et al. (2012); Vu et al. (2015).

## 6 Conclusion

We propose CapsE--a novel embedding model using the capsule network to model relationship triples for knowledge graph completion and search personalization. Experimental results show that our CapsE outperforms other state-of-the-art models on two benchmark datasets WN18RR and FB15k-237 for the knowledge graph completion. We then show the effectiveness of our CapsE for the search personalization, in which CapsE outperforms the competitive baselines on the dataset SEARCH17 of the web search query logs. In addition, our CapsE is capable to effectively model many-to-many relationships. Our code is available at: [https://github.com/daiquocnguyen/CapsE](https://github.com/daiquocnguyen/CapsE).

Figure 4: Learning curves on the validation set with the initial learning rate at \(5e^{-5}\).

 



## Acknowledgements

This research was partially supported by the ARC Discovery Projects DP150100031 and DP160103934. The authors thank Yuval Pinter for assisting us in running his code.

## References

* Bennett et al. (2012) Paul N. Bennett, Ryen W. White, Wei Chu, Susan T. Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long-term behavior on search personalization. In _Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 185-194.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _Journal of Machine Learning Research_, 3:993-1022.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multirelational Data. In _Advances in Neural Information Processing Systems 26_, pages 2787-2795.
* Bordes et al. (2011) Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning Structured Embeddings of Knowledge Bases. In _Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence_, pages 301-306.
* Cai and Wang (2018) Liwei Cai and William Yang Wang. 2018. KBGAN: Adversarial Learning for Knowledge Graph Embeddings. In _Proceedings of The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, page to appear.
* Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In _Proceedings of the 32nd AAAI Conference on Artificial Intelligence_, pages 1811-1818.
* Fox et al. (2005) Steve Fox, Kuldeep Karawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. _ACM Transactions on Information Systems_, 23(2):147-168.
* Harvey et al. (2013) Morgan Harvey, Fabio Crestani, and Mark J. Carman. 2013. Building user profiles from topic models for personalised search. In _Proceedings of the ACM International Conference on Information and Knowledge Management_, pages 2309-2314.
* Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 687-696.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lin et al. (2015a) Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015a. Modeling Relation Paths for Representation Learning of Knowledge Bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 705-714.
* Lin et al. (2015b) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Learning_, pages 2181-2187.
* Nguyen et al. (2018) Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 327-333.
* Nguyen (2017) Dat Quoc Nguyen. 2017. An overview of embedding models of entities and relationships for knowledge base completion. _arXiv preprint_, arXiv:1703.08098.
* Nguyen et al. (2016a) Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. 2016a. Neighborhood Mixture Model for Knowledge Base Completion. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 40-50.
* Nguyen et al. (2016b) Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. 2016b. STransE: a novel embedding model of entities and relationships in knowledge bases. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 460-466.
* Nickel et al. (2016a) Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016a. A Review of Relational Machine Learning for Knowledge Graphs. _Proceedings of the IEEE_, 104(1):11-33.
* Nickel et al. (2016b) Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016b. Holographic Embeddings of Knowledge Graphs. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, pages 1955-1961.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543.
* Pinter and Eisenstein (2018) Yuval Pinter and Jacob Eisenstein. 2018. Predicting Semantic Relations using Global Graph Properties. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1741-1751

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In _Advances in Neural Information Processing Systems_, pages 3859-3869.
* Socher et al. (2013) Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In _Advances in Neural Information Processing Systems 26_, pages 926-934.
* Tevan et al. (2005) Jaime Tevan, Susan T. Dumais, and Eric Horvitz. 2005. Personalizing search via automated analysis of interests and activities. In _Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 449-456.
* Tevan et al. (2011) Jaime Tevan, Daniel J. Liebling, and Gayathri Ravichandran Geetha. 2011. Understanding and predicting personal navigation. In _Proceedings of the ACM International Conference on Web Search and Data Mining_, pages 85-94.
* Tevan et al. (2009) Jaime Tevan, Meredith Ringel Morris, and Steve Bush. 2009. Discovering and using groups to improve personalized search. In _Proceedings of the ACM International Conference on Web Search and Data Mining_, pages 15-24.
* Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015. Observed Versus Latent Features for Knowledge Base and Text Inference. In _Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality_, pages 57-66.
* Trouillon et al. (2016) Theo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In _Proceedings of the 33nd International Conference on Machine Learning_, pages 2071-2080.
* Vu et al. (2017) Thanh Vu, Dat Quoc Nguyen, Mark Johnson, Dawei Song, and Alistair Willis. 2017. Search personalization with embeddings. In _Proceedings of the European Conference on Information Retrieval_, pages 598-604.
* Vu et al. (2015) Thanh Vu, Alistair Willis, Son Ngoc Tran, and Dawei Song. 2015. Temporal latent topic user profiles for search personalisation. In _Proceedings of the European Conference on Information Retrieval_, pages 605-616.
* Wang et al. (2017) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge Graph Embedding: A Survey of Approaches and Applications. _IEEE Transactions on Knowledge and Data Engineering_, 29(12):2724-2743.
* Wang et al. (2014) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. In _Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence_, pages 1112-1119.
* West et al. (2014) Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge Base Completion via Search-based Question Answering. In _Proceedings of the 23rd International Conference on World Wide Web_, pages 515-526.
* White et al. (2013) Ryen W. White, Wei Chu, Ahmed Hassan, Xiaodong He, Yang Song, and Hongning Wang. 2013. Enhancing personalized search by mining and modeling task behavior. In _Proceedings of the World Wide Web conference_, pages 1411-1420.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In _Proceedings of the International Conference on Learning Representations_