Question: What is the baseline model？
Evidence: Link prediction is a subtask of knowledge graph completion to complete a triplet $(h, r, t)$ with $h$ or $t$ missing, i.e., predict $t$ given $(h, r)$ or predict $h$ given $(r, t)$ . Rather than requiring one best answer, this task emphasizes more on ranking a set of candidate entities from the knowledge graph.
Similar to BIBREF2 , we use two measures as our evaluation metrics. (1) Mean Rank: the averaged rank of correct entities or relations; (2) Hits@p: the proportion of valid entities or relations ranked in top $p$ predictions. Here, we set $p=10$ for entities and $p=1$ for relations. A lower Mean Rank and a higher Hits@p should be achieved by a good embedding model. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking (except the test triplet of interest). We call this evaluation setting “Filter”. The evaluation results are reported under these two settings.
We select the margin $\gamma $ among $\lbrace 1, 2\rbrace $ , the embedding dimension $d$ among $\lbrace 20, 50, 100\rbrace $ , the regularization $\eta $ among $\lbrace 0, 1E{-5}, 1E{-6}\rbrace $ , two learning rates $\lambda _s$ and $\lambda _t$ among $\lbrace 0.001, 0.01, 0.05\rbrace $ to learn the parameters of structure and text encoding. The dissimilarity measure is set to either $L_1$ or $\lbrace 1, 2\rbrace $0 distance.
In order to speed up the convergence and avoid overfitting, we initiate the structure embeddings of entity and relation with the results of TransE. The embedding of a word is initialized by averaging the linked entity embeddings whose description include this word. The rest parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .
The final optimal configurations are: $\gamma = 2$ , $d=20$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $\lambda _t = 0.1$ , and $L_1$ distance on WN18; $\gamma =2$ , $d=100$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $d=20$0 , and $d=20$1 distance on FB15K.
Experimental results on both WN18 and FB15k are shown in Table 2 , where we use “Jointly(CBOW)”, “Jointly(LSTM)” and “Jointly(A-LSTM)” to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE.
From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description.
On WN18, the reason why “Jointly(A-LSTM)” is slightly worse than “Jointly(LSTM)” is probably because the number of relations is limited. Therefore, the attention mechanism does not have obvious advantage. On FB15K, “Jointly(A-LSTM)” achieves the best performance and is significantly higher than baseline methods on mean rank.
Although the Hits@10 of our models are worse than the best state-of-the-art method, TransD, it is worth noticing that the score function of our models is based on TransE, not TransD. Our models are compatible with other state-of-the-art knowledge embedding models. We believe that our model can be further improved by adopting the score functions of other state-of-the-art methods, such as TransD.
Besides, textual information largely alleviates the issue of sparsity and our model achieves substantial improvement on Mean Rank comparing with TransD. However, textual information may slightly degrade the representation of frequent entities which have been well-trained. This may be another reason why our Hits@10 is worse than TransD which only utilizes structural information.
For the comparison of Hits@10 of different kinds of relations, we categorized the relationships according to the cardinalities of their head and tail arguments into four classes: 1-to-1, 1-to-many, many-to-1, many-to-many. Mapping properties of relations follows the same rules in BIBREF2 .
Table 3 shows the detailed results by mapping properties of relations on FB15k. We can see that our models outperform baseline TransE in all types of relations (1-to-1, 1-to-N, N-to-1 and N-to-N), especially when (1) predicting “1-to-1” relations and (2) predicting the 1 side for “1-to-N” and “N-to-1” relations.
To get more insights into how the joint representation is influenced by the structure and text information. We observe the activations of gates, which control the balance between two sources of information, to understand the behavior of neurons. We sort the entities by their frequencies and divide them into 50 equal-size groups of different frequencies, and average the values of all gates in each group.
Figure 3 gives the average of gates in ten groups from high- to low-frequency. We observe that the text information play more important role for the low-frequency entities.
From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description.
Answer: TransE.