言語処理学会 第25回年次大会 発表論文集 (2019年3月)

End-to-End Scientific Knowledge Graph Completion via Word
Embedding based Entity Type Classification
Qin Dai1 , Naoya Inoue1,2 , Paul Reisert2 , Kentaro Inui1,2
1
Tohoku University, Japan
2
RIKEN Center for Advanced Intelligence Project, Japan{daiqin, naoya-i, preisert, inui}@ecei.tohoku.ac.jp

Abstract
Knowledge Graph (KG) such as FreeBase (Bollacker et al., 2008) has beenplaying an essential role for numerous
Natural Language Processing (NLP) tasks.
However, the sparsity of existing KG
has stimulated the research in Knowledge
Graph Completion (KGC), which is proposed to predict the missing knowledge
triplet (h, r, t), via projecting KG into
a continuous vector space. Most existing methods are only evaluated on general knowledge triplet e.g., (Tokyo, capitalOf, Japan) rather than scientific knowledge triplet e.g., (hypotension, may betreated by, dopamine). Additionally, exiting method merely collects entity type information from existing knowledge baserather than from semantic feature of entity. In this paper, we investigate the effectiveness of some representative KGC
models on scientific domain and proposeda new KGC model that predicts missing knowledge triplets by incorporating aword embedding based entity type classification model into a state-of-the-art KGCmodel. Our experiments on scientificdataset prove the effectiveness of the proposed model on scientific KGC.

1Introduction

Knowledge Graphs such as Freebase (Bollacker
et al., 2008) and DBpedia (Lehmann et al., 2015)
are extremely crucial for many NLP tasks such as
Question Answering (QA), Information Retrieval(IR), Relation Extraction (RE), etc.
(Schuhmacher and Ponzetto, 2014). They provide large
collections of relations between entities, typicallystored as (h, r, t) triplets, where h = head entity,r = relation and t = tail entity, e.g., (Tokyo, cap-

italOf, Japan). However, the sparsity of KGs impedes their usefulness in real world applications.
Knowledge Graph Completion (KGC) aims toautomatically infer missing facts by examining
the latent regularities in existing KG (Trouillon etal., 2016). One of the most promising method
for capturing the latent regularities is to embed KG into a low-dimensional continuous vector space and estimate the plausibility of potential knowledge triplets via the vector based algebraic calculation. Recent years, various KG embedding models show excellent performance on
KGC, which include TransE (Bordes et al., 2013),
TransD (Ji et al., 2015), ComplEx (Trouillon etal., 2016) and SimplE (Kazemi and Poole, 2018),
etc. However, most KGC models are merely evaluated by general knowledge, such as (Turtle Diary, /film/film/country, United States), rather thanscientific knowledge, such as (pain, may be prevented by, naproxen). For investigating their performance on scientific domain and facilitating scientific KGC, in the paper, we select some representative KGC models and assess their performance on scientific KG.
For enhancing KGC, a variety of external information is potentially useful, among which Entity Type (ET) information, such as ET author
for entity William Shakespeare, has been proved
to be an effective information for KGC (Xie etal., 2016). However, existing approach merelycollects ET information from existing ET knowledge base, rather than applying semantic featureof entity. Distributional representation, typicallylarge corpus trained word embedding, has shownto contain the semantic information about word
categories or types (Yaghoobzadeh and Schütze,2016). For leveraging the effective ET information and simultaneously getting rid of the dependency on an incomplete ET knowledge base,in this work, we proposed an end-to-end KGC
model. The proposed model is capable of iden-― 77 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

tifying ET via the word embedding of target entity
and incorporating the predicted ET into a stateof-to-art KGC model to evaluate the plausibilityof potential knowledge triplets. Our experiments
on scientific KGC show that our end-to-end KGC
model has significant improvements compared toall baselines.

Knowledge
Graph
Completion
Part

KG embedding
Word embedding

KG Scoring

Entity Type

Entity Typee.g.,Disease or Symptom

e.g.,Chemical

Entity TypeClassiﬁcation
Part

MLP

MLP

...

2

We select SimplE (Kazemi and Poole, 2018) as
our base scientific KGC model, since it is simple and strong, achieving state-of-the-art performance in general domain. Specifically, suppose
we have a KG containing a set of knowledge
triplets O = {(e1 , r, e2 )}, where each knowledge
triplet consists of two entities e1 , e2 ∈ E and theirrelation r ∈ R. Here E and R stand for theset of entities and relations respectively. SimplEthen encodes each entity e ∈ E into two vectors
he , te ∈ Rd and each relation r ∈ R into two
vectors vr , vr−1 ∈ Rd respectively, where d isthe dimensionality of the embedding space. hecaptures the entity e’s behaviour as the head entity of a knowledge triplet and te captures e’s behaviour as the tail entity of a knowledge triplet.
vr represents r in triplet (e1 , r, e2 ), while vr−1
represents its inverse relation r−1 in the triplet(e2 , r−1 , e1 ). The KG scoring function of SimplE
for a triplet (e1 , r, e2 ) is defined as φ(e1 , r, e2 ) =1
vr−1 , te1 i). hv, w, xi is de2 (hhe1 , vr , te2 i + hhe2 ,Pdfined as hu, v, wi =n=1 [u]n [v]n [w]n , where[·]n is the n-th entry of a vector. Besides the standard SimplE model, there is a simplified SimplE
called SimplE-ignr, where the scoring function is
simplified as φ(e1 , r, e2 ) = hhe1 , vr , te2 i.

3Proposed Model for Scientific KGC

The proposed end-to-end KGC model is illustrated
in Figure 1. It includes ET classification part (below) and KGC part (above). In ET classification
part, a multi-layer perceptron (MLP) with one hidden layer is applied to identify ET based on wordembedding of target entity. In KGC part, head entity and tail entity along with their predicted ETs
and their relation are projected into corresponding
KG embeddings, which are then fed to a KG scoring function.
3.1

...

Base Model for Scientific KGC

ET Classification Part

In this work, we use a MLP network to classify ET
for head entity and tail entity. The architecture of

Knowledge
Fact

Head Entity

Relation

Tail Entity

e.g.,"hypotension"e.g.,"may be treated by"e.g.,"dopamine"

Figure 1: Overview of the proposed end-to-end
KGC model
our MLP network is as bellow:
h = sigmoid(W xe )(1)

y = sigmoid(V h)(2)where W is a word embedding projection matrix, which is initialized by the pre-trained wordembedding that is trained on scientific corpus via
Gensim word2vec tool, xe is a one-hot entity representation, h is the hidden vector, V is ET projection matrix and y is the output vector containing
the prediction probabilities of all target ETs. W
and V are weight matrices to optimize.
3.2

KGC Part

We extend the base model introduced in Section 2
by incorporating the ET information from the ETclassification part. Specifically, given knowledge
triplet and predicted ET pair ET1 (for e1 ) and ET2(for e2 ), the proposed model project them into
their corresponding KG embeddings namely he1 ,te1 , vr , vr−1 , he2 , te2 , hET1 , tET1 , hET2 and tET2
respectively, where hET1 (or tET1 ) represents theKG embedding of ET for e1 when e1 acts as thehead entity (or tail entity) in a knowledge triplet.
The KG scoring function is defined via Equation 3.
1
φ(e1 , r, e2 ) = (hhe1 , vr , te2 i4+hhe2 , vr−1 , te1 i(3)+hhET1 , vr , tET2 i+hhET2 , vr−1 , tET1 i)

4

Evaluation

4.1

Data

The scientific KG used in this work is extracted from the Unified Medical Language Sys-

― 78 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

#Entity
27,702#Relation
358#Train89,236#Valid5,089#Test
10,239

Table 1: Statistics of scientific KG in this work.
tem (Humphreys et al., 1998) (UMLS), which
includes medical concepts, relations, definitions,etc. As the RE task proposed by (Wang etal., 2014), we only collect the knowledge triplet
with RO categories (RO stands for “has Relationship Other than synonymous, narrower, or
broader”), which covers the interesting relations
like may treat, my prevent, etc.
From the
UMLS 2018 release, we extract about 35 thousand entity-ET pairs (e.g., heart rates-ClinicalAttribute) for training ET classification modeland about 100 thousand knowledge triplets, which
are then randomly divided into training, validationand testing sets for KGC. The statistics of the extracted KG is shown in Table 1. Since we address
the KGC in medical domain, we use the MEDLINE corpus to train the domain specific word embedding.
4.2

Setup

To learn the proposed KGC model, we use AdaGrad with mini-batches. We use the default setting of the base KGC model. Specifically, we
use the initial learning rate of 0.1, mini-batch sizeof 1415 and the embedding dimension of 200
for both entity and relation. We sample 1 negative entity for each ground truth entity and generate a labelled batch LB by labelling positive
triplets as +1 and negatives as −1. We optimize
the L2 regularized negative
log-likelihood of thePbatch, namely minθ ((h,r,t),l)∈LB sof tplus(−l ·φ(h, r, t)) + γkθk22 , where θ represents the parameters and γ = 2.0.
4.3

Result and Discussion

We choose the link prediction task to evaluate theperformance of KGC models. In this task, the
KGC models predict the missing entity, given an
entity and a relation. Specifically, the task predicts tail entity t given both head entity h and relation r, e.g., (h, r, ∗), or predict head entity h given(∗, r, t).
We report the mean reciprocal rank (MRR) forthe evaluated models. Table 2 presents the performance of the selected and proposed KGC models. It can be viewed that the proposed KGC model
does a good job compared with the existing base-lines on the scientific dataset. This proves the effectiveness of the proposed end-to-end architecture for scientific KGC. Specifically, incorporating
word embedding based ET classification modelis an effective approach to improve the performance of KGC. The table also shows that the KGC
models that give state-of-the-art results on generaldomain, such as ComplEx and SimplE, equallyachieve good performance on scientific KGC. This
indicates the domain independency of these KGC
Models, thus, we could apply them into scientific
KGC.

5

Related Work

(Nickel et al., 2011; Bordes et al., 2013; Wang etal., 2014; Ji et al., 2015; Trouillon et al., 2016;
Liu et al., 2017; Kazemi and Poole, 2018) propose
KG embedding models to calculate plausibility ofpotential knowledge triplets using structural information from existing knowledge triplets. Asidefrom the existing knowledge triples, external information is applied to improve KGC. The external information includes surrounding text (Riedel
et al., 2013; Wang et al., 2014; Zhao et al., 2015),
entity type and relation domain (Guo et al., 2015;
Chang et al., 2014; Xie et al., 2016), logical rules(Wang et al., 2015; Rocktäschel et al., 2015) andcross-lingual triples(Klein et al., 2017). However,these methods have not utilized the word embedding based ET classification model for KGC.

6

Conclusion

In this paper, we propose an end-to-end KGC
model to address the scientific KGC. The proposed model could classify ET based on word embedding and leverage the ET information to enhance the KGC performance. Our experiments
not only indicate the domain independency ofthe existing KGC models, but also show ourmodel achieves substantial improvements against
the state-of-the-art baselines.

Acknowledgement
This work was supported by JST CREST Grant
Number JPMJCR1513, Japan and KAKENHI
Grant Number 16H06614.

References
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim
Sturge, and Jamie Taylor. 2008. Freebase: a col-― 79 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

ModelTransE
TransD
ComplEx
SimplE-ignr
SimplE (Base model)
Proposed model

MRR
Raw Filter
0.143 0.175
0.103 0.106
0.117 0.186
0.121 0.186
0.145 0.229
0.200 0.295

1
0.117
0.068
0.115
0.118
0.162
0.222

Hit@
3
0.191
0.111
0.220
0.213
0.259
0.329

10
0.283
0.175
0.324
0.324
0.356
0.430

Table 2: Results on scientific KGC.
laboratively created graph database for structuring
human knowledge. In Proceedings of the 2008 ACM
SIGMOD international conference on Managementof data, pages 1247–1250. AcM.

Maximilian Nickel, Volker Tresp, and Hans-Peter
Kriegel. 2011. A three-way model for collective
learning on multi-relational data. In ICML, volume 11, pages 809–816.

Antoine Bordes, Nicolas Usunier, Alberto GarciaDuran, Jason Weston, and Oksana Yakhnenko.
2013. Translating embeddings for modeling multirelational data. In Advances in neural information
processing systems, pages 2787–2795.

Sebastian Riedel, Limin Yao, Andrew McCallum, andBenjamin M Marlin. 2013. Relation extraction withmatrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
74–84.

Kai-Wei Chang, Scott Wen-tau Yih, Bishan Yang, andChris Meek. 2014. Typed tensor decomposition ofknowledge bases for relation extraction.
Shu Guo, Quan Wang, Bin Wang, Lihong Wang, and
Li Guo. 2015. Semantically smooth knowledge
graph embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:
Long Papers), volume 1, pages 84–94.
Betsy L Humphreys, Donald AB Lindberg, Harold M
Schoolman, and G Octo Barnett. 1998. The unified
medical language system: an informatics researchcollaboration. Journal of the American Medical Informatics Association, 5(1):1–11.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and
Jun Zhao. 2015. Knowledge graph embedding via
dynamic mapping matrix. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 687–696.
Seyed Mehran Kazemi and David Poole. 2018. Simple
embedding for link prediction in knowledge graphs.
Patrick Klein, Simone Paolo Ponzetto, and GoranGlavaš. 2017. Improving neural knowledge basecompletion with cross-lingual projections. Association for Computational Linguistics.
Jens Lehmann, Robert Isele, Max Jakob, Anja
Jentzsch, Dimitris Kontokostas, Pablo N Mendes,
Sebastian Hellmann, Mohamed Morsey, Patrick
Van Kleef, Sören Auer, et al. 2015. Dbpedia–alarge-scale, multilingual knowledge base extracted
from wikipedia. Semantic Web, 6(2):167–195.
Hanxiao Liu, Yuexin Wu, and Yiming Yang. 2017.
Analogical inference for multi-relational embeddings. arXiv preprint arXiv:1705.02426.

Tim Rocktäschel, Sameer Singh, and Sebastian Riedel.
2015. Injecting logical background knowledge into
embeddings for relation extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
1119–1129.
Michael Schuhmacher and Simone Paolo Ponzetto.
2014. Knowledge-based graph document modeling.
In Proceedings of the 7th ACM international conference on Web search and data mining, pages 543–
552. ACM.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric
Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages
2071–2080.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng
Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI, volume 14, pages
1112–1119.
Quan Wang, Bin Wang, Li Guo, et al. 2015. Knowledge base completion using embeddings and rules.
In IJCAI, pages 1859–1866.
Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016.
Representation learning of knowledge graphs withhierarchical types. In IJCAI, pages 2965–2971.
Yadollah Yaghoobzadeh and Hinrich Schütze.
2016.
Corpus-level fine-grained entity typing
using contextual information.
arXiv preprint
arXiv:1606.07901.
Yu Zhao, Zhiyuan Liu, and Maosong Sun. 2015. Representation learning for measuring entity relatedness
with rich information. In IJCAI, pages 1412–1418.

― 80 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.