A Joint Training Framework for Open-World Knowledge Graph Embeddings

1 . Introduction . Knowledge graphs ( KG ) are facts structured in the form of a graph , where the nodes are the entities , and the edges are the relationships between those entities . For example , consider the fact “ Narendra Modi is the Prime Minister of India and lives in India. ” . In this example , Narendra Modi and India are two entities connected by relations isPrimeMinister and livesIn . KGs are often represented in the form of triples , example , ( Narendra Modi , isPrimeMinister , India ) . KG embeddings are continuous multi-dimensional vectors of entities and relations that , through a predefined scoring function , rank real facts in a knowledge base over spurious ones . KG embeddings have a number of applications ranging from KG completion [ Bordes et al. , 2013 ] to natural language inference [ Peters et al. , 2019 ] , knowledge-aware conversation generation [ Zhou et al. , 2018 ] , question-answering [ Saxena et al. , 2020 ] and recommendation [ Chen et al. , 2019 ] . Large knowledge bases like DBpedia [ Auer et al. , 2007 ] rely on unstructured information sources like Wikipedia , which is growing at the rate of 602 new articles per day1 . Applications or tasks that rely on knowledge graph embeddings require approaches to embed new entities introduced due to evolving knowledge bases . Traditional ( or closed-world ) KG embedding models , are limited to the transductive settings , where all the entities and the relations are known during learning . Thus , these approaches are not suitable for KGs where the entities and the relations are augmented over time . An ideal KG embedding model should be able to generalise to newer ( or open-world ) entities and relations in the KGs without re-learning on the updated KGs . We require a method to embed any new ( or open-world ) entities based on the rich semantic information that exist about them in the form of new articles , and also based on the existing connections in the KG . This brings up two tasks that make use of an entity ’ s description- ( 1 ) Open-World KG Completion- which is the task of relating the new entity to the rest of the KG , and ( 2 ) Open-World Entity Embedding- which is the task of producing an embedding for a new entity . Few approaches have been introduced in recent years for these open-world KG related tasks . Studying the literature , we identify five critical properties that make an open-world KG embedding approach efficient , robust and useful for a plethora of applications . 1 . Open-world Embedding Generation : KG applications like question answering or recommendation require entity and relation embeddings . Techniques like ConMask [ Shi and Weninger , 2018 ] which are only optimised for KG Completion and do not produce entity/relation embeddings can not be used for such downstream applications . Hence , it is important that the approaches must also embed open-world KG entities . 2 . Joint Training : To embed open-world entities using their descriptions , we need to align two spaces – description space and KG structure space . Techniques that learn this alignment while keeping one or both of these spaces fixed are said to be offline . Techniques that learn the alignment operation and individual embeddings at the same time are called joint . Studies [ Ormazabal et al. , 2019 ] have shown that embeddings produced by joint training of the two spaces have significantly improved geometric properties as compared to offline training . These geometric properties are important to retain the KG structure on the introduction of new entities , thus , resulting in better performance in downstream applications . 3 . Efficient Ranking : Connecting a new entity to the rest of the KG requires the ranking of potential connections . The score is generated for all possible triples which are ranked to determine the top connections [ Bordes et al. , 2013 , Yang et al. , 2015 , Trouillon et al. , 2016 ] . The scoring function should preferably have time complexity of the order of the embedding dimensions . For real-time applications , the time and memory requirements for inference must be minimal and thus large neural networks models are not efficient as a scoring function to score all possible triples . 4 . Modular : As observed by [ Chang et al. , 2020 ] , different KG embedding approaches are suitable for different applications . For instance , translational methods [ Bordes et al. , 2013 ] are better at clustering and entity classification applications , whereas multiplicative methods [ Trouillon et al. , 2016 , Yang et al. , 2015 ] are better at KG completion . For this reason , our open-world framework must be modular with respect to the choice of the structural 1. https : //en.wikipedia.org/wiki/Wikipedia : Statistics embeddings to accommodate a variety of applications . 5 . Sequence-Size Aware : Any proposed approach must be able to leverage long descriptions using state-of-the-art language models . At the same time , large , popular KGs like Wikidata [ Vrandečić and Krötzsch , 2014 ] have concise sentence-long descriptions that do not benefit from large language models . Any proposed approach must be efficient and perform competitively when run on datasets with short descriptions . Our contributions are summarised below . Our code and datasets are available at https : //github.com/RBC-DSAI-IITM/KnowledgGraphZeroShotLearning . 1 . We propose FOlK - the first-ever joint , modular , efficient and sequence size aware framework to produce open-world KG entity embeddings . 2 . We empirically evaluate performance of our model on open-world KG completion , closed-world KG completion and entity classification . We demonstrate significant improvement of upto 72 % on mean reciprocal rank over the state-of-the-art . 2 . Problem Definition . Before discussing existing approaches and our solution , we give a brief introduction to the problem of learning embeddings for open-world KG entities . Knowledge graphs primarily consist of a set of entities E = { e1 , e2 , ... e|E| } , a set of relations R = { r1 , r2 , ... r|R| } and a set of ( head , relation , tail ) triples T ⊆ { τi : ( hi , ri , ti ) |hi , ti ∈ E , ri ∈ R } . We refer to the set of entities E with relationship information in the existing KG as the closed or structural entity set . In many real world scenarios we often have the descriptions of each entity in our knowledge graph . We denote the description set as D = { d1 , d2 , ... d|E| } . ei and ri are real ( or complex valued ) vectors or embeddings . For model like TransE [ Bordes et al. , 2013 ] or DistMult [ Yang et al. , 2015 ] , ei , ri ∈ Rδ ( where δ denotes the embedding dimension ) . ei , ri ∈ Cδ in the case of models like ComplEx [ Trouillon et al. , 2016 ] . Throughout we use boldface to indicate the vector embedding corresponding to an entity or relation . We use h , r and t to denote head , relation and tail embeddings respectively , in a triple . Entities in a knowledge graph may appear as heads , tails or both . As knowledge bases expand we are presented with a new set of entities E ′ = { e′1 , e ′ 2 , ... e ′ |E ′ | } and their corresponding descriptions D′ = { d′1 , d ′ 2 , ... d ′ |E ′ | } . Our task is to embed the open entity set E ′ using D′ . 3 . Related Work . In this section , we briefly summarise state of the art closed-world or structural KG embedding approaches , approaches with open-world entities and their limitations . Structural KG Embeddings : These models use only the triple or structural information of a KG . Most models are learnt by minimising an energy function on ( h , r , t ) triples through negative sampling . TransE [ Bordes et al. , 2013 ] uses a translational energy function whereas DistMult [ Yang et al. , 2015 ] and ComplEx [ Trouillon et al. , 2016 ] use multiplicative energy functions . The energy and loss functions of all three methods are detailed in Table 8 . Approaches with open-world entities : These models use additional textual descriptions of the entities along with structural information available in KG . ( 1 ) DKRL [ Xie et al. , 2016 ] , constructs two separate spaces - the structural embedding space and the description embedding space and aligns the two using a version of the cross-completion loss we discuss in Section 6.3 . A key limitation of this approach is its inability to generate embeddings for new open-world entities as a consequence of training the two embeddings in separate spaces . ( 2 ) JointE [ Zhao et al. , 2017 ] , uses tf-idf weighted bag-of-words features to encode a description and proposes a custom multiplicative energy function to score a triple . The approach is not modular hence limits the usability of approach for specific applications . Similarly , ( 3 ) ConMask ’ s [ Shi and Weninger , 2018 ] model also uses a custom energy function to score a triple . A triple is scored using a combination of multiple features generated using masked fusion of the entity description with entity and relation names . The usage of a large neural network model to score a triple limits the scalability of the model to score all possible triples , thus filtering methods are used to filter out certain triples resulting in information loss . Further details to be discussed in Sections 6.2 and 6.3 . ( 4 ) OWE , uses offline training to learn a projection matrix to align descriptions to KG structure . The KG embeddings are pre-trained and not modified at alignment time . We discuss the limitations of such an approach in Section 6.3 . We have compared the properties of various approaches and our framework in Table 9 . 4 . Proposed FOlK Model Overall architecture of our framework , FOlK , is presented in Figure 1 . It has three components- ( 1 ) Structural Embedding module to learn the embeddings of the entities and relations already existing in a KG ( 2 ) Description Embedding module which is an encoder to represent the textual information of open-world as well as closed-world entities . ( 3 ) Description Projection module to project the description to a space where the structure and description embeddings are aligned , this represents the KG embedding space used for any of the downstream tasks like KG completion or entity classification . When presented with a new entity , we induce the KG embedding by encoding it ’ s description and projecting it to KG embedding space . The following sections will discuss these modules in detail along with various training strategies that we have used . 4.1 Structural Embeddings . Our framework is agnostic to the choice of the structural KG embedding model . Unless otherwise specified , we use ComplEx as our default method as it has out-performed other structural embedding methods as reported in a recent survey [ Dai et al. , 2020 ] . We briefly explain a few notations first . Let E test and Evalid be the open-world test and validation sets of entities that are not available in the training set . We denote the train , validation and test splits of the triple set T by Ttrain , Tvalid and Ttest . T opentest and T openvalid denote the test and validation sets of triples where either the head or tail of each triple occurs in E test or Evalid respectively . We augment the train split Ttrain with corrupted triplets , i.e , Tcorr ⊆ { τcorr.i : ( hi , ri , ti ) |hi , ti ∈ E , ri ∈ R , ( hi , ri , ti ) /∈ Ttrain } to obtain T ◦ = Ttrain ∪ Tcorr . We use the indicator variable Ih , r , t ∈ { 1 , −1 } to return 1 ( or -1 ) corresponding to whether ( h , r , t ) ∈ Ttrain ( or not ) . We denote the parameters of this module ( the real or complex embeddings ) by ΘS . The loss function used by ComplEx is represented in equation 1 , where Re ( x ) is real part of complex value x and t is complex conjugate . The loss functions of TransE and DistMult are shown in Table 8 . We also include a regularization loss term Lreg which is the average squared ` 2-norm of all the structural embeddings with the regularization parameter λ. L = ∑ ( h , r , t ) ∈T ◦ log ( 1 + exp ( −Ih , r , tRe ( 〈h , r , t〉 ) ) ) ( 1 )

Summary: This paper presents an approach to learn representations of (open-world) entities in a KG with given textual descriptions of open-world entities. This paper describes a technique that jointly learns embeddings for KG entities from descriptions and KG structure for open-world knowledge graph completion. The technique is experimentally validated on the YAGO3-10-Open and WN18RR-Open datasets and it beats previous open world entity representation learning methods.