Neural Concept Formation in Knowledge Graphs

a principled way , and how to effectively exploit them to produce more accurate neural link prediction models . Specifically , we show how concept membership relationships learned via unsupervised clustering of entities can be reified and used to augment a KG . In a thorough set of experiments we confirm that neural link predictors trained on these augmented KGs , or in a joint Expectation-Maximization iterative scheme , can generalize better and produce more accurate predictions for infrequent relationships . For instance , our method yields relative improvements of up to 8.6 % MRR on WN18RR for rare predicates , and up to 82 % in small-data regimes , where the model has access to just a small subset of the training triples . Furthermore , our proposed models are able to learn meaningful concepts . 1 . Introduction . One of the most remarkable aspects of human intelligence is arguably the capacity to abstract and summarize knowledge into concepts . It is believed to play a central role in allowing humans to learn quickly from few examples [ Lake et al. , 2015 ] and to robustly generalize to unseen data [ Pothos and Chater , 2002 , Lakoff and Johnson , 1980 , Rosch et al. , 1976 ] . It is no wonder that many machine learning and knowledge representation methods have tried to “ reverse-engineer ” how humans learn concepts [ Tenenbaum , 2018 , Hassabis et al. , 2017 ] in order to automate reasoning as well as knowledge base construction [ Kok and Domingos , 2007 , Kemp et al. , 2006 , Xu et al. , 2006 ] . Among the most prominent knowledge representation formalisms , there are Knowledge Graphs ( KGs ) – graph-structured knowledge bases where knowledge about the world is encoded in the form of relationships between entities . KGs encode facts about entities and the relationships between them ( edges ) as subject-predicate-object triples , each denoting a relationship of type predicate between the subject and object of the triple . A fundamental task in the construction of KGs is link prediction , which consists of identifying missing edges between entities in the KG . Neural link predictors are a class of link prediction models achieving state-of-the-art results on several link prediction benchmarks while being able to scale to very large KGs . Neural link predictors learn embedding representations for each entity and relation in the KG via back-propagation [ Nickel et al. , 2016 ] . However , neural link predictors are known not to be accurate in the presence of sparse KGs , i.e. , when entities appear only in few triples [ Pujara et al. , 2017 ] , also referred to as the cold-start problem , and may not be able to learn patterns involving groups of entities [ Evans and Grefenstette , 2018 ] . In this work , we propose to learn concepts in neural link predictors as a principled way to elicit discrete latent information that can alleviate the generalization issues of existing models . Moreover , learning new concepts and turning them into entities can help to automate the construction of KGs . Specifically , we make the following contributions . First , we formalize concept learning as an unsupervised clustering step over entities in a KG . We do this first by reifying concept membership relationships into KG facts and by incorporating them in the KG , in a process called KG augmentation , akin to Gad-Elrab et al . [ 2020 ] . Then , we demonstrate that training out-of-the-box neural link predictors on these augmented KGs improves their accuracy . Secondly , we introduce a single , principled probabilistic framework for jointly learning concept memberships and neural link prediction models at once , by maximizing the likelihood of the KG triples via an Expectation-Maximization scheme . Lastly , we execute a rigorous empirical evaluation on several real-world KG benchmarks , showing that both of our approaches , named ConFormA and ConFormAE , are capable of learning semantically meaningful concepts . We observe that explicitly augmenting the KG with the newly-learned concepts can improve generalization over rare predicates by up to 8.6 % in terms of Mean Reciprocal Rank ( MRR ) on WN18RR and 2.1 % on FB15k-237 . Furthermore , we perform a sparsification analysis where neural link predictors are trained on only a small subset of the training set , showing that ConFormA and ConFormAE can achieve up to 82 % relative improvement on WN18RR and 21 % on FB15k-237 when trained on only 5 % of the data . This highlights the potential of our approach for addressing the cold-start problem in sparse KGs . 2 . Background . Let a KG G be represented as a set of N triples , i.e. , G = { 〈s , r , o〉i } Ni=1 ⊆ E × R × E where E = { ei } Nei=1 is the set of subject ( s ) and object ( o ) entities , and R = { ri } Nr i=1 the set of relation types ( r ) . Neural link predictors can be framed as learning a k-dimensional Algorithm 1 ConFormA ( G , Nc , n ) 1 : Input : KG G , no . of clusters Nc , no . of epochs n 2 : Output : Parameters Θ , cluster memberships S = { Sc | c ∈ C } 3 : P← propositionalization ( G ) . E.g. , random paths 4 : S ← Clustering ( P , Nc ) . E.g. , spectral clustering 5 : G′ ← G ∪ { 〈ei , ISA , cj〉 | ei ∈ Sc , c ∈ C } . Create an augmented KG 6 : Θ← init ( ) 7 : for n epochs do . Train the parameters Θ of a neural link predictor on G′ 8 : Θ← train ( G′ , Θ ) return Θ , S representation , i.e. , an embedding vector e ∈ Ck , for all entities in E appearing in G. Given a triple 〈s , r , o〉 ∈ E ×R×E , a neural link predictor defines a scoring function φr : Ck×Ck 7→ R that , given the embedding representations es ∈ Ck and eo ∈ Ck of the subject s and the object o of the triple , returns the score φr ( es , eo ) ∈ R that s and o are related by the relation type r. The scoring function φr implicitly defines a probability distribution p over triples . Let xsro be a binary random variable denoting the existence of the triple 〈s , r , o〉 , then we have that the probability of observing the triple is proportional to its score , i.e. , p ( xsro ) ∝ φr ( es , eo ) [ Nickel et al. , 2016 ] . Examples of neural link predictors are TransE [ Bordes et al. , 2013 ] , DistMult [ Yang et al. , 2015b ] , and ComplEx [ Trouillon et al. , 2016 ] . In this work , we adopt the latter two as baselines , as they achieve state-of-the-art performance on a number of benchmarks when trained with care [ Ruffinelli et al. , 2020 ] . Let E ∈ CNe×k and W ∈ CNr×k be the relation and entity embeddings . Then , learning a neural link predictor from data consists of finding the best set of parameters Θ = { E , W } that solve the optimization problem arg minΘ L ( G ; Θ ) + λΩ ( Θ ) , where L ( G ; Θ ) denotes a modelspecific loss , typically proportional to the negative log-likelihood of the triples [ Trouillon et al. , 2016 ] , and Ω ( · ) is a regularization term , such as the L2 or the nuclear weighted 3-norm [ N3 , Lacroix et al. , 2018 ] , whose weight is determined by a coefficient λ ≥ 0 . In the following section , we discuss how this learning loop for neural link predictors can be extended to learn concepts in a KG , while treating the link predictor model as a black-box . 3 . ConFormA : Learning Concepts by Augmenting Knowledge Graphs Given a KG G , we define concept learning as identifying sets of entities S1 , . . . , SNc ⊆ E that are semantically related and can be abstracted into concepts c1 , . . . , cNc ∈ C. We aim at finding a partitioning of entities S = { Si } Nci=1 , such that each entity is assigned to a single concept at a time , i.e. , ∀Si , Sj ∈ S → Si ∩ Sj = ∅ , and ⋃ S∈S S = E . To this end , a natural solution is to perform a hard clustering of the entities in the KG G. Then , we can reify the cluster membership relations , i.e. , introducing the concepts c1 , . . . , cNc as new entities , and materializing the concept membership relations as new triples . We refer to this process as KG augmentation . Algorithm 1 summarizes our framework , which can be instantiated for different clustering and neural link prediction models . We name it Concept Formation via Augmentation ( ConFormA ) . Next , we discuss how to perform these two steps , and the reasons why neural link predictors can benefit from being trained on augmented KGs . Clustering entities . Ideally , the clustering step in ConFormA could be performed by any relational clustering algorithm . However , classical probabilistic approaches such as statistical predicate invention [ Kok and Domingos , 2007 ] and stochastic block models [ Kemp et al. , 2006 , Xu et al. , 2006 ] would hardly scale to modern KGs with hundreds of thousands of entities . This poses a challenge also to kernel-based approaches [ Blondel et al. , 2008 , de Vries , 2013 , Morris et al. , 2017 ] . To overcome this issue , we opt for a more computationally efficient alternative : we first propositionalize entities into d-dimensional embedding vectors [ Kramer et al. , 2001 ] and then employ a propositional clustering algorithm – e.g. , spectral clustering [ Ng et al. , 2001 ] or K-means – over this now tabular representation P ∈ RNe×d . We find that executing multi-hop random paths in G , as proposed by Das et al . [ 2020 ] , provides scalable and accurate entity representations . We also explored clustering directly over the embeddings learned by a neural link prediction model but with scarce or no link prediction improvements ( see Appendix E , Table 15 ) . This could be due to the latent concept information which we explicitly introduce via augmentation being already captured by the neural link predictor embeddings , while graph-based features add complementary information , as observed in Nickel et al . [ 2016 ] . Knowledge Graph Augmentation . Given the set S , we reify the cluster membership relations by materializing new triples to augment G. Specifically , for each entity e participating in a cluster c ∈ C we create a new triple of the form 〈e , ISA , c〉 , where c is a new entity denoting the j-th concept , and ISA is a freshly introduced relation denoting concept memberships.1 Let G′ denote the augmented KG , i.e. , G′ ← G ∪ { 〈e , ISA , c〉 | e ∈ Sc , c ∈ C } , where Sc is the set of entities assigned to concept c. Learning a neural link predictor simply requires calling its usual training routine ( lines 6-8 of Algorithm 1 ) over the augmented KG G′ . Specifically , its set of parameters Θ′ = { E , W , C } , which now includes the concept embeddings C ∈ CNc×k for the newly introduced concept entities C = { c1 , . . . , cNc } , can be updated by minimizing the neural link predictor ’ s loss function , along its other parameters . ConFormA is likely to improve the generalization of a neural link predictor for the following reasons . Firstly , this kind of augmentation acts as injecting background knowledge that does not need to be learned from scratch , akin to when inverse relation triples [ Lacroix et al. , 2018 , Kazemi and Poole , 2018 ] or hierarchical relation information [ Zhang et al. , 2018 ] are explicitly added to KGs . Secondly , they help to make very sparse KGs more dense , tackling the sparsity issues in neural link predictors [ Pujara et al. , 2017 ] . 4 . ConFormAE : Jointly learning Concepts and Embeddings ConFormA is a flexible framework : it can be customized with any propositionalization and clustering routines , and wrapped around any neural link prediction model . A natural question then arises : is it possible to automatically devise a propositionalization scheme that enhances clustering and embedding quality , that is , to jointly learn both the concepts and the embeddings ? Ideally , we could cast this as a joint optimization problem to maximize the marginal log-likelihood of the triples in G , where marginalization is performed over some latent variable Z denoting the cluster assignments , i.e. , having values c ∈ C. As directly maximizing this marginal likelihood is intractable , we adopt an iterative Expectation-Maximization 1 . A similar augmentation strategy has been independently proposed in Gad-Elrab et al . [ 2020 ] to learn rule-based explanations for a subset of the KG entities . See Section 5 for a discussion . Algorithm 2 ConFormAE ( Θ , S , Nc , n , t ) 1 : Input : no . of clusters Nc , initial parameters Θ , no . of epochs n , and no . of iterations t. 2 : Output : Updated parameters Θ , cluster memberships S = { Sc | c ∈ C } 3 : for t iterations do 4 : for c ∈ C do . Initialise the cluster memberships 5 : Sc ← ∅ 6 : for e ∈ E do 7 : ĉ← arg maxc∈C φISA ( ee , ec ) 8 : Sĉ ← Sĉ ∪ { e } 9 : G′ ← G ∪ { 〈e , ISA , c〉 | e ∈ Sc , c ∈ C } . E-step : Make hard assignments 10 : for n epochs do . M-step : Refine the neural link prediction model 11 : Θ← train ( G′ , Θ ) return Θ , S ( EM ) -like scheme [ Dempster et al. , 1977 ] . Algorithm 2 summarizes the whole process , which we name ConFormAE – Concept Formation with Augmentation via EM . We next discuss in detail how to design the expectation ( E ) and maximization ( M ) steps efficiently . E-step . Let p ( Z | e ) denote the distribution over concept-memberships induced by the neural link predictor for the entity e. Recall from Section 2 that the probability of assigning entity e to concept c is proportional to the score assigned to the reified triple ( Section 3 ) encoding its concept membership , i.e. , p ( Z = c | e ) ∝ φISA ( ee , ec ) , where ee , ec denote the embeddings of e and c , respectively . Exactly computing all the cluster memberships p ( Z = c | e ) for each entity e and concept c ∈ C is a hard problem , since we would need to compute an intractable partition function . We therefore resort to compute hard cluster assignments , a practical approximation commonly adopted in many hard-EM variants [ Samdani et al. , 2012 , Kok and Domingos , 2007 ] . That is , we are interested in solving ĉ = arg maxc∈C p ( Z = c | ei ) for each entity ei . Note that this can be done exactly and efficiently as ĉ = arg maxc∈C φISA ( ee , ec ) and therefore it reduces to predicting the most probable link between the entity e and a concept in C. M-step . The aim of this step is to find the best set of parameters Θ′ for a neural link predictor by maximizing its log-likelihood , i.e. , Ece∼p ( ·|e ) [ log p ( X ) + ∑ e∈E log p ( Z = ce | e ) ] where log p ( X ) here compactly denotes the likelihood of the data in G , and ce ∈ C denotes the concept associated with the entity e ∈ E . This quantity can be efficiently approximated via our reification and augmentation scheme . In fact , at the end of the E-step , we had retrieved the clustering S ( as in ConFormA , cf . Section 3 ) . Therefore , to find Θ′ we can simply train the neural link prediction model for a certain number of epochs n over the augmented KG G′ . We refer to Appendix A for an in-depth analysis of the time and space complexity of ConFormA and ConFormAE .

Summary: The paper aims to improve the performance of neural link predictors on sparse KGs, by augmenting KGs with clusters of concepts. It proposes two versions of augmentations: (1) ConFormA which inserts the clustered concept, and (2) ConformAE which jointly learn the concepts and embeddings through an EM scheme. The authors show that the augmentation helps improve performance of two baselines. on different datasets.