Venue: NAACL 2019

# Graph Pattern Entity Ranking Model

for Knowledge Graph Completion

 Takuma Ebisu\({}^{1,2}\)

Ryutaro Ichise\({}^{2,1,3}\)

\({}^{1}\)SOKENDAI, Chiyoda-ku, Tokyo, Japan

\({}^{2}\)National Institute of Informatics, Chiyoda-ku, Tokyo, Japan

\({}^{3}\)National Institute of Advanced Industrial Science and Technology, Koto-ku, Tokyo, Japan

{takuma,ichise}@nii.ac.jp

###### Abstract

Knowledge graphs have evolved rapidly in recent years and their usefulness has been demonstrated in many artificial intelligence tasks. However, knowledge graphs often have lots of missing facts. To solve this problem, many knowledge graph embedding models have been developed to populate knowledge graphs and these have shown outstanding performance. However, knowledge graph embedding models are so-called black boxes, and the user does not know how the information in a knowledge graph is processed and the models can be difficult to interpret. In this paper, we utilize graph patterns in a knowledge graph to overcome such problems. Our proposed model, the _graph pattern entity ranking model_ (GRank), constructs an entity ranking system for each graph pattern and evaluates them using a ranking measure. By doing so, we can find graph patterns which are useful for predicting facts. Then, we perform link prediction tasks on standard datasets to evaluate our GRank method. We show that our approach outperforms other state-of-the-art approaches such as ComplEx and TorusE for standard metrics such as HITS@_n_ and MRR. Moreover, our model is easily interpretable because the output facts are described by graph patterns.

## 1 Introduction

Knowledge graphs can be used to describe real-world relations as facts in a form that a computer can easily process and has been used for many artificial intelligence tasks Hakimov et al. (2012); Daiber et al. (2013); Bordes et al. (2014). In a knowledge graph, a fact is represented by a labeled and directed edge, called a triple \((h,r,t)\), where \(h\) and \(t\) are entity nodes and \(r\) is a relation label of an edge from \(h\) to \(t\). Knowledge graphs such as YAGO Suchanek et al. (2007), DBpedia Auer et al. (2007), and Freebase Bollacker et al. (2008) have developed rapidly in recent years and are used for many artificial intelligence tasks such as question answering, content tagging, fact-checking, and knowledge inference. Although some knowledge graphs already contain millions of entities and billions of facts, they might still be incomplete and some facts may be missing. Hence, we need to develop a system that can predict missing facts to complete knowledge graphs automatically.

Many kinds of models for _link prediction_ have been developed to estimate unknown facts. Knowledge graph embedding models, which are the most widely used approach in this field, map entities and relations in a knowledge graph onto a vector space and obtain the latent underlying features. However, these models are generally difficult to interpret, as we do not know how information is processed in the models and the predicted facts are output without explanation.

In this paper, we construct statistical models based on _graph pattern matching_. These models are not only easy to interpret compared to knowledge graph embedding models but also outperform state-of-the-art models for link prediction.

Our main contributions in this paper are as follows:

* Defining _graph pattern association rules_ (GPARs) for a knowledge graph.
* Introducing _a graph pattern probability model_ (GPro) and discussing its flaws.
* Proposing a novel model, _the graph pattern entity ranking model_ (GRank), which uses graph patterns to rank entities.
* Proposing distributed rankings to address the problem arising from having the same score for multiple entities

* Evaluating the proposed models through link prediction tasks for standard datasets: It is shown that our model outperforms most state-of-the-art knowledge graph embedding models for the HITS@_n_ and MRR metrics.

The remainder of this paper is organized as follows. In Section 2, we discuss related work on link prediction. In Section 3, we define the terms and notation used in this paper. In Section 4, we define standard confidences for GPARs and discuss their problems. In Section 5, we propose the GRank model to deal with these problems. In Section 6, we present an experimental study in which we compare our models with baseline results for benchmark datasets. In Section 7, we conclude this paper.

## 2 Related Work

We categorize related work for link prediction into two groups: work on knowledge graph embedding models (which are latent feature models) and work on observed feature models.

### Knowledge Graph Embedding Models

Recently, knowledge graph embedding models have yielded great results in link prediction tasks. Knowledge graph embeddings models embed entities and relations on a continuous space and can be roughly classified into three types: translation-based models, bilinear models, and neural network-based models.

The first translation-based model was the TransE Bordes et al. (2013) model, which gained attention because of its effectiveness and simplicity. TransE employs the principle \({\mathbf{h}+\mathbf{r}=\mathbf{t}}\), where \({\mathbf{h},\mathbf{r}}\) and \({\mathbf{t}}\) are the embeddings of \(h,r\) and \(t\), respectively. While this principle efficiently captures first-order rules, the TransE approach still has some problems. The conflict between principle and regularization is one of these problems and the TorusE Ebisu and Ichise (2018) model was recently proposed to solve this problem by embedding entities and relations on a torus manifold.

RESCAL Nickel et al. (2011) was the first bi-linear model, where each relation is represented by a square matrix and the score of the triple \((h,r,t)\) is calculated by a bilinear map which corresponds to the matrix of the relation \(r\) and whose arguments are \({\mathbf{h}}\) and \({\mathbf{t}}\). Hence, RESCAL represents the most general form of a bilinear model. Extensions of RESCAL have been proposed by restricting bilinear functions, for example, DistMult Yang et al. (2015) and ComplEx Trouillon et al. (2016) restrict the matrices representing the relations to diagonal matrices.

Neural network-based models have layers and an activation function like a neural network. The Neural Tensor Network (NTN) Socher et al. (2013) has a standard linear neural network structure and a bilinear tensor structure, and can be considered as a generalization of RESCAL, where the weight of the network is trained for each relation. Graph Convolutional Networks (GCNs) Duvenaud et al. (2015); Defferrard et al. (2016); Kipf and Welling (2017) exploit the convolution operator to capture local information for a graph, however these models are for undirected graphs. Relational GCNs Schlichtrull et al. (2017) and ConvE Dettmers et al. (2018) are generalizations of GCNs for knowledge graphs.

Knowledge graph embedding is the standard approach for link prediction. However, it suffers from low interpretability, resulting in triples which are predicted without any clear reason.

### Observed Feature Models

The main advantage of observed feature models over knowledge graph embedding models is their interpretability. Additionally, Toutanova et al. Toutanova et al. (2015) proposed a relatively simple logistic regression model, the Node+LinkFeat model, which utilizes only one-hop information in a knowledge graph and demonstrated that it performs far better for link prediction on standard datasets than most existing knowledge graph embedding models. However, it has also been shown that the Node+LinkFeat model cannot deal with a low-redundancy dataset because the model uses information which is too local. On the other hand, it has shown that a logistic regression model, the PRA model Lao and Cohen (2010); Lao et al. (2011), which utilizes multi-hop information do not have sufficient accuracy Liu et al. (2016). This suggests logistic regression does not have enough power to deal with deep information. These studies have motivated research toward developing a more efficient model utilizing deeper information.

We begin by discussing GPARs, which were proposed recently by Fan et al. Fan et al. (2015), and have shown their usefulness for social network graphs 

because graph patterns can capture deeper information lying in a knowledge graph and a GPAR explicitly describe the process of prediction. However, the definition of GPARs by Fan et al. cannot be applied to a knowledge graph because Fan et al. assumes a different structure for a social network graph than a knowledge graph. In the following section, we define GPARs for a knowledge graph.

## 3 Preliminaries

In this section, we introduce the definitions and notation required to discuss GPAR-based models.

### Graph Pattern Association Rules on Knowledge Graphs

We modify GPARs for application to a knowledge graph following the definitions of Fan et al. (2015).

**Knowledge Graph:** A _graph_ is defined as \(G=\{(h,r,t)\}\subset E\times R\times E\), where \(E\) denotes a set of entities and \(R\) denotes a set of relations. An element \((h,r,t)\) of \(G\) is called a triple and represents the directed relation \(r\) between \(h\) and \(t\).

An example graph \(G_{ex}\) is shown in Figure 1, where \(p_{i}\) represents a person, Teams "A", "B", and "C" represent sports teams, and countries are entities in \(E_{ex}\) with labeled arrows between two entities representing directed relations in \(R_{ex}\).

**Graph Pattern:** A _graph pattern_ on \(G\) is a graph \(GP_{(x,y)}=\{(z_{i},r,z_{j})\}\subset V_{GP}\times R\times V_{GP}\), where \(V_{GP}\) denotes a set of variables, \(x\) and \(y\) are two designated variables, and \(R\) is the set of relations of \(G\). We suppose \(V_{G}P\) has no redundancy, in other words, \(\forall z\in V_{GP},\exists(z_{i},r,z_{j})\in GP_{(x,y)},z=z_{i}\lor z=z_{j}\).

Some examples of graph patterns on \(G_{ex}\) are shown in Figure 2, where \(GP_{1,(x,y)}=\{(z,\textsf{member\_of},x),(z,\textsf{nationality},y)\}\), \(GP_{2,(x,y)}=\{(z,\textsf{manager\_of},x),(z,\textsf{nationality},y)\}\), and located_in\({}_{(x,y)}=\{(x,\textsf{located\_in},y)\}\). Our focus in this paper is on finding useful graph patterns for link prediction.

**Graph Pattern Matching Function:** A _matching function_ of \(GP_{(x,y)}\) on \((h,t)\in E\times E\) is an injective function \(m:V_{GP}\to E\) that satisfies the following conditions: \(m(x)=h\), \(m(y)=t\), and for all \((z_{i},r,z_{j})\in GP_{(x,y)}\), \((m(z_{i}),r,m(z_{j}))\in G\). \(M(GP_{(x,y)},(h,t))\) denotes the set of all matching functions of \(GP_{(x,y)}\) on \((h,t)\). We say \(GP_{(x,y)}\) matches \((h,t)\) if there is at least one matching function of \(GP_{(x,y)}\) on \((h,t)\) (i.e. \(M(GP_{(x,y)},(h,t))\neq\emptyset\)).

For example, \(m:V_{GP_{1,(x,y)}}\rightarrow E_{ex}\) (\(m(x)=\text{Team A},m(z)=\text{p}_{1},m(y)=\text{U.K.}\)) is a matching function of \(GP_{1,(x,y)}\) on (Team A, U.K.).

**GPAR:** A _graph pattern association rule_ (GPAR) \(AR\) is defined as \(GP_{(x,y)}\Rightarrow r_{(x,y)}\), where \(GP_{(x,y)}\) and \(r_{(x,y)}\) are graph patterns and \(r_{(x,y)}=\{(x,r,y)\}\).

For example, a GPAR \(AR_{1}=P_{1,(x,y)}\Rightarrow\textsf{located\_in}_{(x,y)}\) would indicate that if there is a

Figure 1: Graph \(G_{ex}\) of sports teams.

Figure 2: Graph patterns on \(G_{ex}\).

 

matching function of \(GP_{1,(x,y)}\) on \((h,t)\), then it is likely that there is also a matching function of \(\mathsf{located\_in}_{(x,y)}\) on \((h,t)\), i.e. \((h,\mathsf{located\_in},t)\) is a fact.

### Reconstruction of Knowledge Graph to Queries

Our task is the _link prediction_ of a knowledge graph, i.e. to predict the missing entity of a _query_, which is formally defined as follows:

**Query:** A _query_ is a triple which is missing an entity: \((h,r,?)\) or \((?,r,t)\).

We divide a knowledge graph \(G\) into queries and answers to use as training data for our model. Let \(Q_{r,\mathrm{head}}\) (\(Q_{r,\mathrm{tail}}\)) denote the set of training queries missing a head (tail) entity for a relation \(r\) obtained from \(G\); then \(Q_{r,\mathrm{head}}\) (\(Q_{r,\mathrm{tail}}\)) is defined as follows:

\[Q_{r,\mathrm{head}}=\{(?,r,t)\mid(h,r,t)\in G\},\] \[Q_{r,\mathrm{tail}}=\{(h,r,?)\mid(h,r,t)\in G\}\]

In this case, the answers of training queries are defined as follows:

\[a_{(?,r,t)}=\{h\mid(h,r,t)\in G\},\] \[a_{(h,r,?)}=\{t\mid(h,r,t)\in G\}\]

A knowledge graph usually contains only positive triples. Hence, we adopt the _partial completeness assumption_ (PCA) Galarraga et al. (2013, 2015) to generate negative answers.

**Partial Completeness Assumption:** if \((h,r,t)\) is in \(G\), then

\[\forall t^{\prime}\in E,((h,r,t^{\prime})\notin G\Rightarrow(h, r,t^{\prime})\text{ is negative}) \tag{1}\] \[\forall h^{\prime}\in E,((h^{\prime},r,t)\notin G\Rightarrow(h^{ \prime},r,t)\text{ is negative}) \tag{2}\]

The standard PCA definition consists only of Equation (1), but we add Equation (2) because we also need to allow negative answers for \(Q_{r,\mathrm{head}}\). Under PCA, negative answers for each question are defined as follows:

\[n_{(?,r,t)}=E\setminus a_{(?,r,t)},~{}n_{(h,r,?)}=E\setminus a_{(h,r,?)}\]

## 4 Standard Confidence and Problems

### AMIE with GPARs

An association rule is essentially a binary classifier, i.e. the antecedent of an association rule matches or does not match, and an association rule is thus evaluated. Following this idea, we suggest the most straightforward way to define the _confidence_, which indicates the reliability of an association rule, is the conditional probability, which is the probability of the consequent given the antecedent for a GPAR. The conditional probability \(Pr_{tail}(r_{(x,y)}\mid GP_{(x,y)})\) of a GPAR \(GP_{(x,y)}\Rightarrow r_{(x,y)}\) to predict a tail is defined as follows:

\[conf_{tail}(GP_{(x,y)}\Rightarrow r_{(x,y)})=Pr_{tail}(r_{(x,y)}\mid GP_{(x, y)})=\frac{\sum_{(h,r,?)\in Q_{r,\mathrm{tail}}}|\{t\in a_{(h,r,?)}\mid M(GP_{( x,y)},(h,t))\neq\emptyset\}|}{\sum_{(h,r,?)\in Q_{r,\mathrm{tail}}}||\{t^{\prime}\in E \mid M(GP_{(x,y)},(h,t^{\prime}))\neq\emptyset\}|}\]

For each query, the candidate entities found by the graph pattern are counted for the denominator while only correct entities are counted for the numerator. This confidence is used to evaluate GPARs only to answer queries with a missing tail because \(Q_{r,\mathrm{tail}}\) and its answers are used to define it.

Interestingly, GPARs with this confidence are equivalent to AMIE Galarraga et al. (2013, 2015), which was proposed to find horn clauses for a knowledge graph, although AMIE was proposed before the appearance of GPARs. However, AMIE originally has only one confidence value for a GPAR because AMIE is not designed for link prediction. Hence, we introduce the following alternative definition for the confidence value to answer a query missing a head entity.

### Standard Link Prediction Model and Problems

We define another confidence to deal with a query with a missing head entity as follows:

\[conf_{head}(GP_{(x,y)}\Rightarrow r_{(x,y)})=Pr_{head}(r_{(x,y)}\mid GP_{( x,y)})=\frac{\sum_{(?,r,t)\in Q_{r,\mathrm{head}}}|\{h\in a_{(?,r,t)}\mid M(GP_{(x,y)},(h,t))\neq \emptyset\}|}{\sum_{(?,r,t)\in Q_{r,\mathrm{head}}}|\{h^{\prime}\in E\mid M( GP_{(x,y)},(h^{\prime},t))\neq\emptyset\}|}\]

Additionally, we restrict matching functions to injective functions as defined in Section 3.1, which is different from AMIE, because the restriction avoids redundant matching functions which map multiple variables to the same entity and gives a good bias for real-world knowledge. For example, an GPAR \(GP_{3,(x,y)}\Rightarrow\mathsf{sibling\_of}_{(x,y)}\), where \(GP_{3,(x,y)}=\{(z,\mathsf{parent\_of},x),(y,\mathsf{child\_of},z)\}\), is helpful to predict siblings. However, let \(p\) represent a person, \(GP_{3,(x,y)}\) matches \((p,p)\) although \(p\) is not a sibling of \(p\). The above restriction omits such concerns. For another example, a GPAR \(\{(z_{1},\mathsf{manager\_of},x),(z_{1},\mathsf{manager\_of},z_{2})\), \((z_{2},\mathsf{located\_in},y)\}\Rightarrow\mathsf{located\_in}_{(x,y)}\) on the graph 

\(G_{ex}\) in Figure 1 should not be considered helpful because \(m(x)=m(z_{2})\) holds for a matching function \(m\) of the antecedent pattern and as a result, the GPAR is almost tautological. We consider two confidence values for GPARs, \(conf_{fail}\) and \(conf_{head}\), referred to as the _graph pattern probability model_ (GPPro).

However, GPro cannot deal with queries where counting the number of matching functions is crucial. An example where the number of matching functions is important is shown in Figure 1. In \(G_{ex}\), the country that Team C is located in is missing. One might guess that Team C is located in Italy because most of the Team C players have Italian nationality and the nationality of a player often matches the country that the team is located in. However, GPro underestimates the GPAR \(AR_{1,(x,y)}=GP_{1,(x,y)}\Rightarrow\mathsf{located\_in}_{(x,y)}\), which is equivalent to one's guessing process: \(conf_{tail}(AR_{1,(x,y)})=2/5\), while \(conf_{tail}(AR_{2,(x,y)})=1/2\), where \(AR_{2,(x,y)}=GP_{2,(x,y)}\Rightarrow\mathsf{located\_in}_{(x,y)}\). Hence, GPro judges \(AR_{2,(x,y)}\) is more useful than \(AR_{1,(x,y)}\), and as a result, GPro predicts Team C is located in Germany rather than Italy.

This problem is caused by considering a GPAR as a binary classifier, i.e. the matching number is not taken into account. For example, if we apply \(AR_{1,(x,y)}=P_{1,(x,y)}\Rightarrow\mathsf{located\_in}_{(x,y)}\) to a query (Team A,\(\mathsf{located\_in}\),?) in the traditional way (as a binary classifier), the output will contain two entities with equal weighting, the U.K. and France, because \(P_{1,(x,y)}\) matches (Team A,U.K.) and (Team A,France). Then, one of the output entities is correct and the other is incorrect. This is the reason why \(AR_{1,(x,y)}\) is underestimated.

To deal with this problem, in this paper, we consider a GPAR as an entity ranking system by counting the number of matching functions of the antecedent graph pattern rather than considering as a binary classifier.

## 5 GPR as Entity Ranking System and Evaluation Metrics

As well as considering a GPAR as a binary classifier, we consider it as an entity ranking system. Entities are ranked according to a score, based on their number of matching functions.

Moreover, we introduce the _distributed rankings_ for entities, which are proposed to deal with situations where multiple entities have the same score. Then, we define the evaluation metrics for the distributed rankings to evaluate GPARs for link prediction.

These approaches overcome the problems shown in Section 4.2.

### GPAR as Entity Ranking System

We consider a GPAR as a ranking system in this section to rank queries for which counting the number of matching functions of the antecedent is helpful, as shown in Section 4.2.

First, we define a scoring function whose arguments are a graph pattern \(GP_{(x,y)}\) and a pair of entities \((h,t)\). The scoring function returns the number of matching functions of a pattern on a pair, which is formally defined as follows:

\[score(GP_{(x,y)},(h,t))=|M(GP_{(x,y)},(h,t))|\]

Given a pattern \(GP_{(x,y)}\) and a query \((h,r,?)\), we can obtain the \(score(GP_{(x,y)},(h,t^{\prime}))\) for each candidate tail entity \(t^{\prime}\). Then we obtain the rankings of the tail entities in descending order of the scores. The head entity rankings for a query \((?,r,t)\) are also obtained in this way. This ranking method gives us new perspective when we apply GPARs to answer a query. For example, if we apply \(AR_{1,(x,y)}=P_{1,(x,y)}\Rightarrow\mathsf{located\_in}_{(x,y)}\) to a query (Team A,\(\mathsf{located\_in}\),?) the U.K. will be ranked first and France second. In this situation, we can say that \(AR_{1,(x,y)}\) works because the correct entity ranks higher than the wrong entity. We can basically evaluate a GPAR as an entity ranking system by evaluating output rankings by an evaluation metric for an ranking system such as the _mean average precision_. However, often multiple entities have the same score and traditional metrics cannot deal with the situation. To deal with this problem, we propose a new concept, called _distributed rankings_, and the corresponding metrics in the following sections.

### Distributed Rankings

We propose distributed rankings where each entity can distribute over multiple ranks and each rank can have multiple entities, to deal with situations where multiple entities have the same score.

Traditional rankings of entities are represented by a matrix \(Rank=(rank_{i,j})\in\{0,1\}^{n\times n}\), where \(n\) is the number of entities, and for each column and row there is one 1 element. In this matrix, columns represent entities and rows represent ranks. For example, \(rank_{i,j}=1\) means that the entity \(j\) has 

rank \(i\). On the other hand, distributed rankings of entities are represented by a matrix \(dRank=(drank_{i,j})\in[0,1]^{p\times n}\), where the summation of a column or a row is equal to 1. Different from traditional rankings, the value of each element is continuous and multiple elements can be greater than 0 in a column or a row. For example, \(rank_{i,j}=0.5\) means that half of the entity \(j\) has rank \(i\). Note that a traditional ranking matrix is a distributed ranking matrix.

Given a pattern \(GP_{(x,y)}\) and a query \((h,r,?)\), We obtain distributed rankings of entities, \(dRANK(GP_{(x,y)},(h,r,?))\), according to their scores as follows. Let \(a\) be the number of entities whose scores are greater than the entity represented by \(j\) and let \(b\) be the number of entities whose scores are the same as the entity represented by \(j\). Then, \(drank_{i,j}\), an element of \(dRANK(GP_{(x,y)},(h,r,?))\), is determined to be \(1/b\) for \(a+1\leqq i\leqq a+b\) and 0 otherwise. Distributed rankings of head entities for a query \((?,r,t)\) are obtained in the same way, and we refer to them as \(dRANK(GP_{(x,y)},(?,r,t))\). Unlike traditional rankings, distributed rankings are uniquely determined from the scores of entities.

Traditional rankings can be evaluated by metrics such as the average precision or the cumulative gain. However, distributed rankings cannot be evaluated by these metrics. Hence, we require a different evaluation metric for distributed rankings.

### Evaluation of GPARs as Entity Ranking System

We use a GPAR to obtain distributed entity rankings as shown in Section 5.1. In this section, we define a metric to evaluate distributed rankings of entities by generalizing the _average precision_ to evaluate a GPAR.

For a pattern \(GP_{(x,y)}\) and a training query \((h,r,?)\), the _distributed precision at \(k\)_, \(dPre_{k}\), of \(dRANK(GP_{(x,y)},(h,r,?))\) is defined as follows:

\[dpre_{k}(GP_{(x,y)},(h,r,?))\] \[=\frac{\sum_{i=1}^{k}\sum_{l_{j}\in ab_{k,r}}drank_{i,j}}{k}\]

where \(t_{j}\) is an entity represented by \(j\) and \(drank_{i,j}\) is an element of \(dRANK(GP_{(x,y)},(h,r,?))\). The elements related with correct entities ranked higher or equal to \(k\) are summed up as the traditional precision at \(k\).

Then, the _distributed average precision_, _MAP_, is defined for a pattern \(GP_{(x,y)}\) and a training query \((h,r,?)\) as follows:

\[dAP(GP_{(x,y)},(h,r,?))\] \[=\frac{\sum_{l_{j}\in ab_{(h,r,?)}}\sum_{k=1}^{n}dPre_{k}(GP_{(x, y)},(h,r,?))\times drank_{k,j}}{|a_{(h,r,?)}|}\]

where \(t_{j}\) is an entity represented by \(j\), \(drank_{i,j}\) is an element of \(dRANK(GP_{(x,y)},(h,r,?))\), and \(n\) is the number of entities. The numerator of the average precision for traditional rankings is the summation of the precision at \(k\) for relevant entities. However, a relevant entity represented by \(j\) is distributed over multiple ranks in \(dRANK\) so that the precision at \(k\) multiplied by \(drank_{k,j}\) is summed over \(k\) where a relevant entity \(j\) is distributed. \(dAP(GP_{(x,y)},(?,r,t))\) for a training query with a missing head can be defined in the same way. The _distributed mean average precision_ for a GPAR \(GP_{(x,y)}\Rightarrow r_{(x,y)}\) is defined as follows:

\[dMAP_{head}(GP_{(x,y)}\Rightarrow r_{(x,y)})\] \[=\sum_{(?,r,!)\in Q_{(x,y)}}\frac{dAP(GP_{(x,y)},(?,r,t))}{|Q_{ r,\text{heal}}|}\] \[dMAP_{tail}(GP_{(x,y)}\Rightarrow r_{(x,y)})\] \[=\sum_{(h,r,?)\in Q_{(x,\text{rail}}}\frac{dAP(GP_{(x,y)*},(h,r,? ))}{|Q_{r,\text{tail}}|}\]

We also define dMAP with for the "_filtered_" (Bordes et al., 2013) rankings which are obtained from original rankings by eliminating entities whose corresponding triples (except the target triple) were included in the training dataset. "Filtered" dMAP (fdMAP) is the mean of the dAP of "filtered" rankings for each answer of queries.

We refer to GPARs considered as entity ranking systems with these dMAPs or fdMAPs as the _graph pattern entity ranking model_ (GRank).

By using a graph pattern to rank entities, GRank is able to properly estimate GPARs where the number of matches is important as

\begin{table}
\begin{tabular}{c|c c c c c} Dataset & \# Entities & \# Relations & \# Training & \# Validation & \# Test \\ WN18 & 40.943 & 18 & 141.442 & 5,000 & 5,000 \\ WN1RR & 40.943 & 11 & 86.835 & 3,034 & 3,134 \\ FB15k & 14.951 & 1,345 & 483.142 & 50,000 & 59.071 \\ FB15k-237 & 14,541 & 237 & 272.115 & 17,535 & 20,466 \\ \end{tabular}
\end{table}
Table 1: Statistics of benchmark datasets. The numbers of entities, relations, training triples, validation triples, and test triples are shown.

 

shown in Section 4.2, unlike GPro. For example, \(dMAP_{tail}(AR_{1,(x,y)})=1\), is the maximum value, while \(dMAP_{tail}(AR_{2,(x,y)})=1/2\) in Figure 1. Hence, GRank can answer the query (Team C, located_in,?) by applying \(AR_{1,(x,y)}\).

## 6 Experiments

Our proposed models, GPro (Section 4.2) and GRank (Section 5), are evaluated through link prediction tasks and compared with other state-of-the-art link prediction models.

### Datasets

Experiments were conducted on four benchmark datasets: WN18, FB15k [3], WN18RR [7], and FB15k-237 [9] (details of these datasets are provided in Table 1). These datasets have been widely used in previous studies for evaluating model performance in link prediction tasks.

WN18 and FB15k were extracted from the real knowledge graphs WordNet [10] and Freebase [11], respectively. WordNet is a well-known human-curated lexical database, and hence, WN18 is an easy benchmark of link prediction because it is well constructed and there are few missing or wrong facts. Therefore, link prediction models should perform well on WN18. Freebase is a huge knowledge graph of general facts and there are many missing facts. It is known that WN18 and FB15k have redundancy in the form of reverse relations. For this reason, when WN18RR and FB15k-237 are extracted from WN18 and FB15k, the inverse relations of other relations are removed.

### Evaluation Protocol

We conducted the link prediction task following the same approach reported in [3] to evaluate our models qualitatively and quantitatively. For each test triple \((h_{t},r_{t},t_{t})\) in a dataset, two queries, \((h_{t},r_{t},?)\) and \((?,r_{t},t_{t})\), were constructed in the same way as in Section 3.2. Then, we obtained the rankings of entities for each query from each model as outlined in the following paragraphs. The rankings were "filtered" by eliminating entities whose corresponding triples (except the target test triple) were included in the training, validation, or test dataset. The obtained rankings were scored by their mean reciprocal rank (MRR) and HITS@_n_, where MRR is the mean of the inverse of the ranks of corresponding entities and HITS@_n_ is the proportion of test queries whose corresponding entities are ranked in the top \(n\) of the obtained rankings.

Next, we describe how to obtain rankings from models. We restricted antecedent graph patterns of GPARs to _connected_ and _closed_[1, 1] patterns whose size \(|GP_{(x,y)}|\) was less than or equal to \(L\) to restrict the search space. A connected and closed patterns is a pattern connecting \(x\) and \(y\) without branches, as shown in Figure 2. \(L\) was chosen for each model among \(\{1,2,3\}\) by MRR from the validation triples of each dataset. It took about four days to evaluate all candidate GPARs for GRank with dMAPs in FB15k using 

an _Intel Xeon Gold 6154_ (3.00 GHz, 18 cores).

We now explain how we obtained the rankings for queries with missing heads. For each relation \(r\), we chose the top 1,000 GPARs in descending order of the standard confidence, the dMAP, or the fdMAP to predict the heads. Let \(GP_{i,(x,y)}\Rightarrow r_{(x,y)}\) be the obtained GPAR, where \(i\) denotes the rank. We defined the ordering for two entities for query \((?,r_{t},t_{t})\) as follows: for entities \(e_{1}\) and \(e_{2}\), we define \(e_{1}>e_{2}\) if there exists \(i^{\prime}\) for which \(score(GP_{i,(x,y)},(e_{1},t_{t}))=score(GP_{i,(x,y)},(e_{2},t_{t}))\) for \(i>i^{\prime}\) and \(score(GP_{i^{\prime},(x,y)},(e_{1},t_{t}))>score(GP_{i^{\prime},(x,y)},(e_{ 2},t_{t}))\). We obtained the entity rankings with this ordering for each query. Rankings for queries with missing tails were obtained in the same way.

### Results

The results of the link prediction tasks for our proposed models, GPro, GRank with dMAP, and GRank with fdMAP, are shown in Tables 2, where the results reported in previous studies are included for comparison.

In Table 2, the first seven models are knowledge graph embedding models and the following two models are observed feature models. Table 2 shows the effectiveness of the Node+LinkFeat model Toutanova and Chen (2015), although this model is very simple (high MRRs imply that the model also has high HITS@1s or HITS@3s). The Node+LinkFeat model performed well on WN18 and FB15k because these datasets often contain the reverse relations of other relations. In other words, it shows that knowledge graph embedding models failed to capture this redundancy. On the other hand, our proposed models, GPro and GRank, generally yield better results than the knowledge graph embedding models and results which are better than or comparable to Node+LinkFeat, which means that our models can also handle such redundancy. In particular, GRank with dMAP and fdMAP yielded the best results on FB15k. This indicates that taking the multiplicity of matchings and deeper information into account is important for knowledge graphs such as FreeBase that contain miscellaneous relations and are not well curated like WordNet. As a result, GRank performed well.

Table 2 also shows GPro and GRank yield better results for the WN18RR dataset than the other models. For FB15k-237, the performance of Node+LinkFeat is comparable with most of the other more sophisticated knowledge graph models and GPro does not yield good results because FB15k-237 has less redundancy. GRank also performs better than the most other models for the FB15k-237 dataset for the same reason as the FB15k dataset. However, our models do not utilize the information related to the co-occurrence of entities and relations in triples (node features Toutanova and Chen (2015)), while ConvE, Node+LinkFeat, and other models do. We also limited the size and the shapes of graph patterns because of the calculation time; we will address these and improve our models further in our future work.

Quality of Obtained PathsThe examples of antecedent patterns ranked high by GRank with _dMAP\({}_{tail}\)_ for FB15k are shown in Figure 3. The patterns shown for predicting the sibling relation are all correct as the antecedents of GPARs; however, the MAP of \(GP^{\prime}_{2,(x,y)}\) and \(GP^{\prime}_{3,(x,y)}\) are low. The reason for this is that \(GP^{\prime}_{2,(x,y)}\) works when an individual has more than two siblings. The MAP of \(GP^{\prime}_{3,(x,y)}\) is low because individual's parents are often missing in FB15k. However, they are still ranked higher than other patterns.

The produces_film relation is the inverse relation of the executive_produced_by relation in FB15k. Such patterns are very helpful when performing link prediction tasks, and GRank is able to find them. However, the MAP is not as high because of missing facts. GRank is able to use majority rules such as \(GP^{\prime}_{5,(x,y)}\Rightarrow\text{film\_produced\_by}(\mathsf{x},\mathsf{ y})\) instead in such cases. This rule can be interpreted as stating that a particular film was likely to have been produced by a person who produced many films in the same production company.

Output triples of GRank (and GPAR-based models) are described by antecedent patterns unlike knowledge graph embedding models as shown here.

## 7 Conclusions

In this paper, we first defined GPARs for a knowledge graph and the standard confidence measures of GPARs for link prediction. Then, we pointed out the problems with the standard confidence measures and we introduced a new perspective using GPARs to rank entities to overcome these problems. We also proposed distributed rank 

ings for situations where multiple entities have the same scores and defined metrics for them. This idea led us to propose the GRank model. GRank is easy to interpret because outputs are described by GPARs, unlike knowledge graph embedding models, and so efficient that it outperformed the state-of-the-art knowledge graph embedding models in link prediction tasks.

In future work, we will extend GRank to use more complex patterns. We considered only antecedent graph patterns whose sizes were less than or equal to 3.If we allow antecedent graph patterns to have larger sizes, then we may find more useful GPARs. We also restricted graph patterns to contain only variables and not constants. Hence, we did not use all of the available information contained in the knowledge graph. We believe that using such complex graph patterns will improve GRank further.

## Acknowledgements

This work was partially supported by the New Energy and Industrial Technology Development Organization (NEDO).

We would like to thank Patrik Schneider for helpful writing advice.

## References

* Auer et al. (2007) Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBpedia: A nucleus for a web of open data. In _The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference_, pages 722-735.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_, pages 1247-1250.
* Bordes et al. (2014) Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 615-620.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Advances in Neural Information Processing Systems_, pages 2787-2795.
* Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In _Proceedings of the 9th International Conference on Semantic Systems_, pages 121-124.
* Defferrard et al. (2016) Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In _Advances in Neural Information Processing Systems 29_, pages 3844-3852.
* Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Steentorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_.
* Duvenaud et al. (2015) David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In _Advances in Neural Information Processing Systems 28_, pages 2224-2232.
* Ebisu and Ichise (2018) Takuma Ebisu and Ryutaro Ichise. 2018. Toruse: Knowledge graph embedding on a lie group. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_.
* Fan et al. (2015) Wenfei Fan, Xin Wang, Yinghui Wu, and Jingbo Xu. 2015. Association rules with graph patterns. _VLDB J._, 8(12):1502-1513.

Figure 3: Examples of antecedent patterns for \(dMAP_{tail}\) which were given high ranks by GRank for the FB15k dataset.

 

Luis Galarraga, Christina Telfioudi, Katja Hose, and Fabian M. Suchanek. 2015. Fast rule mining in ontological knowledge bases with AMIE+. _VLDB J._, 24(6):707-730.
* Galarraga et al. [2013] Luis Antonio Galarraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. 2013. AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In _22nd International World Wide Web Conference_, pages 413-422.
* Hakimov et al. [2012] Sherzod Hakimov, Salih Atilay Oto, and Erdogan Dogdu. 2012. Named entity recognition and disambiguation using linked data and graph-based centrality scoring. In _Proceedings of the 4th International Workshop on Semantic Web Information Management_, SWIM '12, pages 1-7.
* Kipf and Welling [2017] Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In _Proceedings of the Fifth International Conference on Learning Representations_.
* Lao and Cohen [2010] Ni Lao and William W. Cohen. 2010. Relational retrieval using a combination of path-constrained random walks. _Machine Learning_, 81(1):53-67.
* Lao et al. [2011] Ni Lao, Tom M. Mitchell, and William W. Cohen. 2011. Random walk inference and learning in A large scale knowledge base. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 529-539.
* Liu et al. [2016] Qiao Liu, Liuyi Jiang, Minghao Han, Yao Liu, and Zhiguang Qin. 2016. Hierarchical random walk inference in knowledge graphs. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 445-454.
* Miller [1995] George A. Miller. 1995. Wordnet: A lexical database for English. _Commun. ACM_, 38(11):39-41.
* Nickel et al. [2016] Maximilian Nickel, Lorenzo Rosasco, and Tomasao A. Poggio. 2016. Holographic embeddings of knowledge graphs. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, pages 1955-1961.
* Nickel et al. [2011] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In _Proceedings of the 28th International Conference on Machine Learning_, pages 809-816.
* Schlichtkrull et al. [2017] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. _CoRR_, abs/1703.06103.
* Socher et al. [2013] Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In _Advances in Neural Information Processing Systems_, pages 926-934.
* Suchanek et al. [2007]Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In _Proceedings of the 16th International Conference on World Wide Web_, pages 697-706.
* Toutanova and Chen [2015] Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In _Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality_.
* Trouillon et al. [2016] Theo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In _Proceedings of the 33rd International Conference on Machine Learning_, pages 2071-2080.
* Yang et al. [2015] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In _Proceedings of the Third International Conference on Learning Representations_