Knowledge Graph Representation with Jointly Structural and Textual Encoding

Abstract: The objective of knowledge graph embedding is to encode both entities and relations of knowledge graphs into continuous low-dimensional vector spaces. Previously, most works focused on symbolic representation of knowledge graph with structure information, which can not handle new entities or entities with few facts well. In this paper, we propose a novel deep architecture to utilize both structural and textual information of entities. Specifically, we introduce three neural models to encode the valuable information from text description of entity, among which an attentive model can select related information as needed. Then, a gating mechanism is applied to integrate representations of structure and text into a unified architecture. Experiments show that our models outperform baseline by margin on link prediction and triplet classification tasks. Source codes of this paper will be available on Github.

Knowledge graphs have been proved to benefit many artificial intelligence applications, such as relation extraction, question answering and so on. A knowledge graph consists of multi-relational data, having entities as nodes and relations as edges. An instance of fact is represented as a triplet (Head Entity, Relation, Tail Entity), where the Relation indicates a relationship between these two entities. In the past decades, great progress has been made in building large scale knowledge graphs, such as WordNet BIBREF0 , Freebase BIBREF1 . However, most of them have been built either collaboratively or semi-automatically and as a result, they often suffer from incompleteness and sparseness.
The knowledge graph completion is to predict relations between entities based on existing triplets in a knowledge graph. Recently, a new powerful paradigm has been proposed to encode every element (entity or relation) of a knowledge graph into a low-dimensional vector space BIBREF2 , BIBREF3 . The representations of entities and relations are obtained by minimizing a global loss function involving all entities and relations. Therefore, we can do reasoning over knowledge graphs through algebraic computations.
Although existing methods have good capability to learn knowledge graph embeddings, it remains challenging for entities with few or no facts BIBREF4 . To solve the issue of KB sparsity, many methods have been proposed to learn knowledge graph embeddings by utilizing related text information BIBREF5 , BIBREF6 , BIBREF7 . These methods learn joint embedding of entities, relations, and words (or phrases, sentences) into the same vector space. However, there are still three problems to be solved. (1) The combination methods of the structural and textual representations are not well studied in these methods, in which two kinds of representations are merely aligned on word level or separate loss function. (2) The text description may represent an entity from various aspects, and various relations only focus on fractional aspects of the description. A good encoder should select the information from text in accordance with certain contexts of relations. Figure 1 illustrates the fact that not all information provided in its description are useful to predict the linked entities given a specific relation. (3) Intuitively, entities with many facts depend more on well-trained structured representation while those with few or no facts might be largely determined by text descriptions. A good representation should learn the most valuable information by balancing both sides.
In this paper, we propose a new deep architecture to learn the knowledge representation by utilizing the existing text descriptions of entities. Specifically, we learn a joint representation of each entity from two information sources: one is structure information, and another is its text description. The joint representation is the combination of the structure and text representations with a gating mechanism. The gate decides how much information from the structure or text representation will carry over to the final joint representation. In addition, we also introduce an attention mechanism to select the most related information from text description under different contexts. Experimental results on link prediction and triplet classification show that our joint models can handle the sparsity problem well and outperform the baseline method on all metrics with a large margin.
Our contributions in this paper are summarized as follows.
In this section, we briefly introduce the background knowledge about the knowledge graph embedding.
Knowledge graph embedding aims to model multi-relational data (entities and relations) into a continuous low-dimensional vector space. Given a pair of entities $(h,t)$ and their relation $r$ , we can represent them with a triple $(h,r,t)$ . A score function $f(h,r, t)$ is defined to model the correctness of the triple $(h,r,t)$ , thus to distinguish whether two entities $h$ and $t$ are in a certain relationship $r$ . $f(h,r, t)$ should be larger for a golden triplet $(h, r, t)$ that corresponds to a true fact in real world, otherwise $r$0 should be lower for an negative triplet.
The difference among the existing methods varies between linear BIBREF2 , BIBREF8 and nonlinear BIBREF3 score functions in the low-dimensional vector space.
Among these methods, TransE BIBREF2 is a simple and effective approach, which learns the vector embeddings for both entities and relationships. Its basic idea is that the relationship between two entities is supposed to correspond to a translation between the embeddings of entities, that is, $\textbf {h}+ \mathbf {r}\approx \mathbf {t}$ when $(h,r,t)$ holds.
TransE's score function is defined as: 
$$f(h,r,t)) &= -\Vert \textbf {h}+\mathbf {r}-\mathbf {t}\Vert _{2}^2$$   (Eq. 5) 
 where $\textbf {h},\mathbf {t},\mathbf {r}\in \mathbb {R}^d$ are embeddings of $h,t,r$ respectively, and satisfy $\Vert \textbf {h}\Vert ^2_2=\Vert \mathbf {t}\Vert ^2_2=1$ . The $\textbf {h}, \mathbf {r}, \mathbf {t}$ are indexed by a lookup table respectively.
Given an entity in most of the existing knowledge bases, there is always an available corresponding text description with valuable semantic information for this entity, which can provide beneficial supplement for entity representation.
To encode the representation of a entity from its text description, we need to encode the variable-length sentence to a fixed-length vector. There are several kinds of neural models used in sentence modeling. These models generally consist of a projection layer that maps words, sub-word units or n-grams to vector representations (often trained beforehand with unsupervised methods), and then combine them with the different architectures of neural networks, such as neural bag-of-words (NBOW), recurrent neural network (RNN) BIBREF9 , BIBREF10 , BIBREF11 and convolutional neural network (CNN) BIBREF12 , BIBREF13 .
In this paper, we use three encoders (NBOW, LSTM and attentive LSTM) to model the text descriptions.
A simple and intuitive method is the neural bag-of-words (NBOW) model, in which the representation of text can be generated by summing up its constituent word representations.
We denote the text description as word sequence $x_{1:n} = x_1,\cdots ,x_n$ , where $x_i$ is the word at position $i$ . The NBOW encoder is 
$$\mathrm {enc_1}(x_{1:n}) = \sum _{i=1}^{n} \mathbf {x}_i,$$   (Eq. 7) 
 where $\mathbf {x}_i \in \mathbb {R}^d$ is the word embedding of $x_i$ .
To address some of the modelling issues with NBOW, we consider using a bidirectional long short-term memory network (LSTM) BIBREF14 , BIBREF15 to model the text description.
LSTM was proposed by BIBREF16 to specifically address this issue of learning long-term dependencies BIBREF17 , BIBREF18 , BIBREF16 in RNN. The LSTM maintains a separate memory cell inside it that updates and exposes its content only when deemed necessary.
Bidirectional LSTM (BLSTM) can be regarded as two separate LSTMs with different directions. One LSTM models the text description from left to right, and another LSTM models text description from right to left respectively. We define the outputs of two LSTM at time step $i$ are $\overrightarrow{\mathbf {z}}_i$ and $\overleftarrow{\mathbf {z}}_i$ respectively.
The combined output of BLSTM at position $i$ is ${\mathbf {z}_i} = \overrightarrow{\mathbf {z}}_i \oplus \overleftarrow{\mathbf {z}}_i$ , where $\oplus $ denotes the concatenation operation.
The LSTM encoder combines all the outputs $\mathbf {z}_i \in \mathbb {R}^d$ of BLSTM at different position. 
$$\mathrm {enc_2}(x_{1:n}) = \sum _{i=1}^{n} {\mathbf {z}_i}.$$   (Eq. 9) 
While the LSTM encoder has richer capacity than NBOW, it produces the same representation for the entire text description regardless of its contexts. However, the text description may present an entity from various aspects, and various relations only focus on fractional aspects of the description. This phenomenon also occurs in structure embedding for an entity BIBREF8 , BIBREF19 .
Given a relation for an entity, not all of words/phrases in its text description are useful to model a specific fact. Some of them may be important for the given relation, but may be useless for other relations. Therefore, we introduce an attention mechanism BIBREF20 to utilize an attention-based encoder that constructs contextual text encodings according to different relations.
For each position $i$ of the text description, the attention for a given relation $r$ is defined as $\alpha _i(r)$ , which is 
$$e_i(r) &= \mathbf {v}_a^T \tanh (\mathbf {W}_a {\mathbf {z}}_i + \mathbf {U}_a \mathbf {r}), \\
\alpha _i(r)&=\operatorname{\mathbf {softmax}}(e_i(r))\nonumber \\
&=\frac{\exp (e_i(r))}{\sum ^{n}_{j=1} \exp (e_j(r))},$$   (Eq. 12) 
 where $\mathbf {r}\in \mathbb {R}^d$ is the relation embedding; ${\mathbf {z}}_i \in \mathbb {R}^d$ is the output of BLSTM at position $i$ ; $\mathbf {W}_a,\mathbf {U}_a \in \mathbb {R}^{d\times d}$ are parameters matrices; $\mathbf {v}_a \in \mathbb {R}^{d}$ is a parameter vector.
The attention $\alpha _i(r)$ is interpreted as the degree to which the network attends to partial representation $\mathbf {z}_{i}$ for given relation $r$ .
The contextual encoding of text description can be formed by a weighted sum of the encoding $\mathbf {z}_{i}$ with attention. 
$$\mathbf {enc_3}(x_{1:n};r) &= \sum _{i=1}^{n} \alpha _i(r) * \mathbf {z}_i.$$   (Eq. 13) 
Since both the structure and text description provide valuable information for an entity , we wish to integrate all these information into a joint representation.
We propose a united model to learn a joint representation of both structure and text information. The whole model can be end-to-end trained.
For an entity $e$ , we denote $\mathbf {e}_s$ to be its embedding of structure information, $\mathbf {e}_d$ to be encoding of its text descriptions. The main concern is how to combine $\mathbf {e}_s$ and $\mathbf {e}_d$ .
To integrate two kinds of representations of entities, we use gating mechanism to decide how much the joint representation depends on structure or text.
The joint representation $\mathbf {e}$ is a linear interpolation between the $\mathbf {e}_s$ and $\mathbf {e}_d$ . 
$$\mathbf {e}= \textbf {g}_e \odot \mathbf {e}_s + (1-\textbf {g}_e)\odot \mathbf {e}_d,$$   (Eq. 14) 
 where $\textbf {g}_e$ is a gate to balance two sources information and its elements are in $[0,1]$ , and $\odot $ is an element-wise multiplication. Intuitively, when the gate is close to 0, the joint representation is forced to ignore the structure information and is the text representation only.
We use the contrastive max-margin criterion BIBREF2 , BIBREF3 to train our model. Intuitively, the max-margin criterion provides an alternative to probabilistic, likelihood-based estimation methods by concentrating directly on the robustness of the decision boundary of a model BIBREF23 . The main idea is that each triplet $(h,r,t)$ coming from the training corpus should receives a higher score than a triplet in which one of the elements is replaced with a random elements.
We assume that there are $n_t$ triplets in training set and denote the $i$ th triplet by $(h_i, r_i, t_i),(i = 1, 2, \cdots ,n_t)$ . Each triplet has a label $y_i$ to indicate the triplet is positive ( $y_i = 1$ ) or negative ( $y_i = 0$ ).
Then the golden and negative triplets are denoted by $\mathcal {D} = \lbrace (h_j, r_j, t_j) | y_j = 1\rbrace $ and $\mathcal {\hat{D}} = \lbrace (h_j, r_j, t_j) | y_j = 0\rbrace $ , respectively. The positive example are the triplets from training dataset, and the negative examples are generated as follows: $ \mathcal {\hat{D}} = \lbrace (h_l, r_k, t_k) | h_l \ne h_k \wedge y_k = 1\rbrace \cup \lbrace (h_k, r_k, t_l) | t_l \ne t_k \wedge y_k = 1\rbrace \cup \lbrace (h_k, r_l, t_k) | r_l \ne r_k \wedge y_k = 1\rbrace $ . The sampling strategy is Bernoulli distribution described in BIBREF8 . Let the set of all parameters be $\Theta $ , we minimize the following objective: 
$$J(\Theta )=\sum _{(h,r,t) \in \mathcal {D}}\sum _{( \hat{h},\hat{r},\hat{t}) \in \mathcal {\hat{D}}} \max \left(0,\gamma - \right. \nonumber \\
f( h,r,t)+f(\hat{h},\hat{r},\hat{t})\left.\right)+ \eta \Vert \Theta \Vert _2^2,$$   (Eq. 22) 
where $\gamma > 0$ is a margin between golden triplets and negative triplets., $f(h, r, t)$ is the score function. We use the standard $L_2$ regularization of all the parameters, weighted by the hyperparameter $\eta $ .
In this section, we study the empirical performance of our proposed models on two benchmark tasks: triplet classification and link prediction.
We use two popular knowledge bases: WordNet BIBREF0 and Freebase BIBREF1 in this paper. Specifically, we use WN18 (a subset of WordNet) BIBREF24 and FB15K (a subset of Freebase) BIBREF2 since their text descriptions are easily publicly available. Table 1 lists statistics of the two datasets.
Link prediction is a subtask of knowledge graph completion to complete a triplet $(h, r, t)$ with $h$ or $t$ missing, i.e., predict $t$ given $(h, r)$ or predict $h$ given $(r, t)$ . Rather than requiring one best answer, this task emphasizes more on ranking a set of candidate entities from the knowledge graph.
Similar to BIBREF2 , we use two measures as our evaluation metrics. (1) Mean Rank: the averaged rank of correct entities or relations; (2) Hits@p: the proportion of valid entities or relations ranked in top $p$ predictions. Here, we set $p=10$ for entities and $p=1$ for relations. A lower Mean Rank and a higher Hits@p should be achieved by a good embedding model. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking (except the test triplet of interest). We call this evaluation setting “Filter”. The evaluation results are reported under these two settings.
We select the margin $\gamma $ among $\lbrace 1, 2\rbrace $ , the embedding dimension $d$ among $\lbrace 20, 50, 100\rbrace $ , the regularization $\eta $ among $\lbrace 0, 1E{-5}, 1E{-6}\rbrace $ , two learning rates $\lambda _s$ and $\lambda _t$ among $\lbrace 0.001, 0.01, 0.05\rbrace $ to learn the parameters of structure and text encoding. The dissimilarity measure is set to either $L_1$ or $\lbrace 1, 2\rbrace $0 distance.
In order to speed up the convergence and avoid overfitting, we initiate the structure embeddings of entity and relation with the results of TransE. The embedding of a word is initialized by averaging the linked entity embeddings whose description include this word. The rest parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .
The final optimal configurations are: $\gamma = 2$ , $d=20$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $\lambda _t = 0.1$ , and $L_1$ distance on WN18; $\gamma =2$ , $d=100$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $d=20$0 , and $d=20$1 distance on FB15K.
Experimental results on both WN18 and FB15k are shown in Table 2 , where we use “Jointly(CBOW)”, “Jointly(LSTM)” and “Jointly(A-LSTM)” to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE.
From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description.
On WN18, the reason why “Jointly(A-LSTM)” is slightly worse than “Jointly(LSTM)” is probably because the number of relations is limited. Therefore, the attention mechanism does not have obvious advantage. On FB15K, “Jointly(A-LSTM)” achieves the best performance and is significantly higher than baseline methods on mean rank.
Although the Hits@10 of our models are worse than the best state-of-the-art method, TransD, it is worth noticing that the score function of our models is based on TransE, not TransD. Our models are compatible with other state-of-the-art knowledge embedding models. We believe that our model can be further improved by adopting the score functions of other state-of-the-art methods, such as TransD.
Besides, textual information largely alleviates the issue of sparsity and our model achieves substantial improvement on Mean Rank comparing with TransD. However, textual information may slightly degrade the representation of frequent entities which have been well-trained. This may be another reason why our Hits@10 is worse than TransD which only utilizes structural information.
For the comparison of Hits@10 of different kinds of relations, we categorized the relationships according to the cardinalities of their head and tail arguments into four classes: 1-to-1, 1-to-many, many-to-1, many-to-many. Mapping properties of relations follows the same rules in BIBREF2 .
Table 3 shows the detailed results by mapping properties of relations on FB15k. We can see that our models outperform baseline TransE in all types of relations (1-to-1, 1-to-N, N-to-1 and N-to-N), especially when (1) predicting “1-to-1” relations and (2) predicting the 1 side for “1-to-N” and “N-to-1” relations.
To get more insights into how the joint representation is influenced by the structure and text information. We observe the activations of gates, which control the balance between two sources of information, to understand the behavior of neurons. We sort the entities by their frequencies and divide them into 50 equal-size groups of different frequencies, and average the values of all gates in each group.
Figure 3 gives the average of gates in ten groups from high- to low-frequency. We observe that the text information play more important role for the low-frequency entities.
Triplet classification is a binary classification task, which aims to judge whether a given triplet $(h, r, t)$ is a correct fact or not. Since our used test sets (WN18 and FB15K) only contain correct triplets, we construct negative triplets following the same setting used in BIBREF3 .
For triplets classification, we set a threshold $\delta _r$ for each relation $r$ . $\delta _r$ is obtained by maximizing the classification accuracies on the valid set. For a given triplet $(h, r, t)$ , if its score is larger than $\delta _r$ , it will be classified as positive, otherwise negative.
Table 4 shows the evaluation results of triplets classification. The results reveal that our joint encoding models is effective and also outperform the baseline method.
On WN18, “Jointly(A-LSTM)” achieves the best performance, and the “Jointly(LSTM)” is slightly worse than “Jointly(A-LSTM)”. The reason is that the number of relations is relatively small. Therefore, the attention mechanism does not show obvious advantage. On FB15K, the classification accuracy of “Jointly(A-LSTM)” achieves 91.5%, which is the best and significantly higher than that of state-of-the-art methods.
Recently, it has gained lots of interests to jointly learn the embeddings of knowledge graph and text information. There are several methods using textual information to help KG representation learning.
 BIBREF3 represent an entity as the average of its word embeddings in entity name, allowing the sharing of textual information located in similar entity names.
 BIBREF5 jointly embed knowledge and text into the same space by aligning the entity name and its Wikipedia anchor, which brings promising improvements to the accuracy of predicting facts. BIBREF6 extend the joint model and aligns knowledge and words in the entity descriptions. However, these two works align the two kinds of embeddings on word level, which can lose some semantic information on phrase or sentence level.
 BIBREF25 also represent entities with entity names or the average of word embeddings in descriptions. However, their use of descriptions neglects word orders, and the use of entity names struggles with ambiguity. BIBREF7 jointly learn knowledge graph embeddings with entity descriptions. They use continuous bag-of-words and convolutional neural network to encode semantics of entity descriptions. However, they separate the objective functions into two energy functions of structure-based and description-based representations. BIBREF26 embeds both entity and relation embeddings by taking KG and text into consideration using CNN. To utilize both representations, they need further estimate an optimum weight coefficients to combine them together in the specific tasks.
Besides entity representation, there are also a lot of works BIBREF27 , BIBREF28 , BIBREF29 to map textual relations and knowledge base relations to the same vector space and obtained substantial improvements.
While releasing the current paper we discovered a paper by BIBREF30 proposing a similar model with attention mechanism which is evaluated on link prediction and triplet classification. However, our work encodes text description as a whole without explicit segmentation of sentences, which breaks the order and coherence among sentences.
We propose a united representation for knowledge graph, utilizing both structure and text description information of the entities. Experiments show that our proposed jointly representation learning with gating mechanism is effective, which benefits to modeling the meaning of an entity.
In the future, we will consider the following research directions to improve our model:

Figure and table captions:
Figure 2: Our general architecture of jointly structural and textual encoding.
Table 1: Statistics of datasets used in experiments.
Table 2: Results on link prediction.
Table 3: Detailed results by category of relationship on FB15K.
Figure 3: Detailed results by entity frequency on FB15K.
Figure 4: Visualization of gates with different entity frequencies on FB15K.
Table 4: Results on triplet classification.

Questions about this paper:
What neural models are used to encode the text?

Answers:
NBOW, LSTM, attentive LSTM