Neural Variational Inference For Embedding Knowledge Graphs

1 INTRODUCTION . In many fields , including physics and biology , being able to represent uncertainty is of crucial importance ( Ghahramani , 2015 ) . For instance , when link prediction in Knowledge Graphs is used for driving expensive pharmaceutical experiments ( Bean et al. , 2017 ) , it would be beneficial to know what is the confidence of a model in its predictions . However , a significant shortcoming of current neural link prediction models ( Dettmers et al. , 2017 ; Trouillon et al. , 2016a ) – and for the vast majority of neural representation learning approaches – is their inability to express a notion of uncertainty . Furthermore , Knowledge Graphs can be very large and web-scale ( Dong et al. , 2014 ) and often suffer from incompleteness and sparsity ( Dong et al. , 2014 ) . In a generative probabilistic model , we could leverage the variance in model parameters and predictions for finding which facts to sample during training , in an Active Learning setting ( Kapoor et al. , 2007 ; Gal et al. , 2017 ) . ( Gal & Ghahramani , 2016 ) use dropout for modelling uncertainty , however , this is only applied at test time . However , current neural link prediction models typically only return point estimates of parameters and predictions ( Nickel et al. , 2016 ) , and are trained discriminatively rather than generatively : they aim at predicting one variable of interest conditioned on all the others , rather than accurately representing the relationships between different variables ( Ng & Jordan , 2001 ) , however , ( Gal & Ghahramani , 2016 ) could still be applied to get uncertainty estimates for these models . The main argument of this article is that there is a lack of methods for quantifying predictive uncertainty in a knowledge graph embedding representation , which can only be utilised using probabilistic modelling , as well as a lack of expressiveness under fixed-point representations . This constitutes a significant contribution to the existing literature because we introduce a framework for creating a family of highly scalable probabilistic models for knowledge graph representation , in a field where there has been a lack of this . We do this in the context of recent advances in variational inference , allowing the use of any prior distribution that permits a re-parametrisation trick , as well as any scoring function which permits maximum likelihood estimation of the parameters . 2 BACKGROUND . In this work , we focus on models for predicting missing links in large , multi-relational networks such as FREEBASE . In the literature , this problem is referred to as link prediction . We specifically focus on knowledge graphs , i.e. , graph-structured knowledge bases where factual information is stored in the form of relationships between entities . Link prediction in knowledge graphs is also known as knowledge base population . We refer to Nickel et al . ( 2016 ) for a recent survey on approaches to this problem . A knowledge graph G , { ( r , a1 , a2 ) } ⊆ R × E × E can be formalised as a set of triples ( facts ) consisting of a relation type r ∈ R and two entities a1 , a2 ∈ E , respectively referred to as the subject and the object of the triple . Each triple ( r , a1 , a2 ) encodes a relationship of type r between a1 and a2 , represented by the fact r ( a1 , a2 ) . Link prediction in knowledge graphs is often simplified to a learning to rank problem , where the objective is to find a score or ranking function φΘr : E × E 7→ R for a relation r that can be used for ranking triples according to the likelihood that the corresponding facts hold true . 2.1 NEURAL LINK PREDICTION . Recently , a specific class of link predictors received a growing interest ( Nickel et al. , 2016 ) . These predictors can be understood as multi-layer neural networks . Given a triple x = ( s , r , o ) , the associated score φΘr ( s , o ) is given by a neural network architecture encompassing an encoding layer and a scoring layer . In the encoding layer , the subject and object entities s and o are mapped to low-dimensional vector representations ( embeddings ) hs , h ( s ) ∈ Rk and ho , h ( o ) ∈ Rk , produced by an encoder hΓ : E → Rk with parameters Γ . Similarly , relations r are mapped to hr , h ( r ) ∈ Rk . This layer can be pre-trained ( Vylomova et al. , 2016 ) or , more commonly , learnt from data by back-propagating the link prediction error to the encoding layer ( Bordes et al. , 2013a ; Nickel et al. , 2016 ; Trouillon et al. , 2016a ) . The scoring layer captures the interaction between the entity and relation representations hs , ho and hr are scored by a function φΘ ( hs , ho , hr ) , parametrised by Θ . Other work encodes the entity-pair in one vector ( Riedel et al. , 2013 ) . Summarising , the high-level architecture is defined as : hs , ho , hr , h Γ ( s ) , hΓ ( o ) , hΓ ( r ) φ ( s , o , r ) , φΘ ( hs , ho , hr ) Ideally , more likely triples should be associated with higher scores , while less likely triples should be associated with lower scores . While the literature has produced a multitude of encoding and scoring strategies , for brevity we overview only a small subset of these . However , we point out that our method makes no further assumptions about the network architecture other than the existence of an argument encoding layer . 2.2 ENCODING LAYER . Given an entity e ∈ E , the entity encoder hΓ is usually implemented as a simple embedding layer hΓ ( e ) , [ Γ ] e , where Γ is an embedding matrix ( Nickel et al. , 2016 ) . For pre-trained embeddings , the embedding matrix is fixed . Note that other encoding mechanisms are conceivable , such as ; recurrent , graph convolution ( Kipf & Welling , 2016a ; b ) or convolutional neural networks ( Dettmers et al. , 2017 ) . 2.3 DECODING LAYER : SCORING FUNCTIONS . DistMult DISTMULT ( Yang et al. , 2015 ) represents each relation r using a parameter vector Θ ∈ Rk , and scores a link of type r between ( hs , ho , hr ) using the following scoring function : φΘ ( hs , ho , hr ) , 〈hs , ho , hr〉 , k∑ i=1 hs , iho , ihr , i , where 〈· , · , ·〉 denotes the tri-linear dot product . ComplEx COMPLEX ( Trouillon et al. , 2016a ) is an extension of DISTMULT using complex-valued embeddings while retaining the mathematical definition of the dot product . In this model , the scoring function is defined as follows : φΘ ( hr , hs , ho ) , Re ( 〈hr , hs , ho〉 ) , , < Re ( es ) , Re ( er ) , Re ( eo ) > + < Im ( es ) , Re ( er ) , Im ( eo ) > + < Re ( es ) , Im ( er ) , Im ( eo ) > − < Im ( es ) , Im ( er ) , Re ( eo ) > where hs , ho , hr ∈ Ck are complex vectors , x denotes the complex conjugate of x , Re ( x ) ∈ Rk denotes the real part of x and Im ( x ) ∈ Ck denotes the imaginary part of x . 3 GENERATIVE MODELS . Let D , { ( τ1 , y1 ) , . . . , ( τn , yn ) } denote a set of labelled triples , where τi , 〈si , pi , oi〉 , and yi ∈ { 0 , 1 } denotes the corresponding label , denoting that the fact encoded by the triple is either true or false . We can assume D is generated by a corresponding generative model . In the following , we propose two alternative generative models . 3.1 LATENT FACT MODEL . In Figure 1 ’ s graphical model , we assume that the Knowledge Graph was generated according to the following generative model . Let V , E ×R× E the space of possible triples . where τ , 〈s , p , o〉 , and hτ , [ hs , hp , ho ] denotes the sampled embedding representations of s , o ∈ E and p ∈ R. Note that , in this model , the embeddings are sampled for each triple . As a consequence , the set of latent variables in this model isH , { hτ | τ ∈ E ×R× E } . The joint probability of the variables pθ ( H , D ) is defined as follows : pθ ( H , D ) , ∏ ( τ , yτ ) ∈D pθ ( hτ ) p θ ( yτ | hτ ) ( 1 ) The marginal distribution over D is then bounded as follows , with respect to our variational distribution q : pθ ( D ) ≥ Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ( 2 ) Proposition 1 As a consequence , the log-marginal likelihood of the data , under the Latent Fact Model , is bounded by : log pθ ( D ) ≤ ∑ ( τ , yτ ) ∈D Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] , ELBO ( 3 ) Proof . We refer the reader to the Appendix 6 for a detailed proof LFM ’ s ELBO . Assumptions : LFM model assumes each fact of is a randomly generated variable , as well as a mean field variational distribution and that each training example is independently distributed . 3.1.1 OPTIMISING THE ELBO . Note that this is an enormous sum over |D| elements . However , this can be approximated via Importance Sampling , or Bernoulli Sampling ( Botev et al. , 2017 ) . ELBO = ∑ ( τ , yτ ) ∈D Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] = ( ∑ ( τ , yτ ) ∈D+ Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ) + ( ∑ ( τ , yτ ) ∈D− Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ) ( 4 ) By using Bernoulli Sampling , ELBO can be approximated by : ELBO ≈ ∑ ( τ , yτ ) ∈D sτ bτ ( Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ) . ( 5 ) where pθ ( sτ = 1 ) = bτ can be defined as the probability that for the coefficient sτ each positive or negative fact τ is equal to one ( i.e is included in the ELBO summation ) . The exact ELBO can be recovered from setting bτ = 1.0 for all τ . We can define a probability distribution of sampling from D+ and D− – similarly to Bayesian Personalised Ranking ( Rendle et al. , 2009 ) , we sample one negative triple for each positive one — we use a constant probability for each element depending on whether it is in the positive or negative set . Proposition 2 The Latent Fact models ELBO can be estimated similarly using a constant probability for positive or negative samples , we end up with the following estimate : ELBO ≈ ( ∑ ( τ , yτ ) ∈D+ sτ b+ ( Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ) ) + ( ∑ ( τ , yτ ) ∈D− sτ b− ( Eqφ [ log pθ ( yτ | hτ ) ] −KL [ qφ ( hτ ) || pθ ( hτ ) ] ) ) ( 6 ) where b+ = |D+|/|D+| and b− = |D+|/|D−| .

Summary: This work proposed two variational embedding methods for knowledge graph inference tasks. The experiments show slight improvements compared to other variational embedding methods, e.g., KG2E, TransG, and slight improvement on the WN18 dataset compared with the non-variational method. On the other hand, both the embedding method and the training of variational models used in this work are already well developed. Thus, this work doesn’t show too much novel contribution. However, the reviewer really appreciate the visualization provided in Fig. 4.