Tensor Decompositions for Temporal Knowledge Base Completion

1 INTRODUCTION . Link prediction in relational data has been the subject of interest , given the widespread availability of such data and the breadth of its use in bioinformatics ( Zitnik et al. , 2018 ) , recommender systems ( Koren et al. , 2009 ) or Knowledge Base completion ( Nickel et al. , 2016a ) . Relational data is often temporal , for example , the action of buying an item or watching a movie is associated to a timestamp . Some medicines might not have the same adverse side effects depending on the subject ’ s age . The task of temporal link prediction is to find missing links in graphs at precise points in time . In this work , we study temporal link prediction through the lens of temporal knowledge base completion , which provides varied benchmarks both in terms of the underlying data they represent , but also in terms of scale . A knowledge base is a set of facts ( subject , predicate , object ) about the world that are known to be true . Link prediction in a knowledge base amounts to answer incomplete queries of the form ( subject , predicate , ? ) by providing an accurate ranking of potential objects . In temporal knowledge bases , these facts have some temporal metadata attached . For example , facts might only hold for a certain time interval , in which case they will be annotated as such . Other facts might be event that happened at a certain point in time . Temporal link prediction amounts to answering queries of the form ( subject , predicate , ? , timestamp ) . For example , we expect the ranking of queries ( USA , president , ? , timestamp ) to vary with the timestamps . As tensor factorization methods have proved successful for Knowledge Base Completion ( Nickel et al. , 2016a ; Trouillon et al. , 2016 ; Lacroix et al. , 2018 ) , we express our Temporal Knowledge Base Completion problem as an order 4 tensor completion problem . That is , timestamps are discretized and used to index a 4-th mode in the binary tensor holding ( subject , predicate , object , timestamps ) facts . First , we introduce a ComplEx ( Trouillon et al. , 2016 ) decomposition of this order 4 tensor , and link it with previous work on temporal Knowledge Base completion . This decomposition yields embeddings for each timestamps . A natural prior is for these timestamps representation to evolve slowly over time . We are able to introduce this prior as a regularizer for which the optimum is a ∗Université Paris-Est , Equipe Imagine , LIGM ( UMR8049 ) Ecole des Ponts ParisTech , Marne-la-Vallée variation on the nuclear p-norm . In order to deal with heterogeneous temporal knowledge bases where a significant amount of relations might be non-temporal , we add a non-temporal component to our decomposition . Experiments on available benchmarks show that our method outperforms the state of the art for similar number of parameters . We run additional experiments for larger , regularized models and obtain improvements of up to 0.07 absolute Mean Reciprocal Rank ( MRR ) . Finally , we propose a dataset of 400k entities , based on Wikidata , with 7M train triples , of which 10 % contain temporal validity information . This dataset is larger than usual benchmarks in the Knowledge Base completion community and could help bridge the gap between the method designed and the envisaged web-scale applications . 2 RELATED WORK . Matrices and tensors are upper case letters . The i-th row of U is denoted by ui while it ’ s j − th column is denoted by U : ,j . The tensor product of two vectors is written ⊗ and the hadamard ( elementwise ) product ⊙ . Static link prediction methods Standard tensor decomposition methods have lead to good results ( Yang et al. , 2014 ; Trouillon et al. , 2016 ; Lacroix et al. , 2018 ; Balažević et al. , 2019 ) in Knowledge Base completion . The Canonical Polyadic ( CP ) Decomposition ( Hitchcock , 1927 ) is the tensor equivalent to the low-rank decomposition of a matrix . A tensor X of canonical rank R can be written as : X = R∑ r=1 U : ,r⊗V : ,r⊗W : ,r = [ [ U , V , W ] ] ⇐⇒ ∀ ( i , j , k ) , Xi , j , k = R∑ r=1 ui , rvj , rwk , r = ⟨ui , vj , wk⟩ Setting U = W leads to the Distmult ( Yang et al. , 2014 ) model , which has been successful , despite only being able to represent symmetric score functions . In order to keep the parameter sharing scheme but go beyond symmetric relations , Trouillon et al . ( 2016 ) use complex parameters and set W to the complex conjugate of U , U . Regularizing this algorithm with the variational form of the tensor nuclear norm as well as a slight transformation to the learning objective ( also proposed in Kazemi & Poole ( 2018 ) ) leads to state of the art results in Lacroix et al . ( 2018 ) . Other methods are not directly inspired from classical tensor decompositions . For example , TransE ( Bordes et al. , 2013 ) models the score as a distance of the translated subject to an object representation . This method has lead to many variations ( Ji et al. , 2015 ; Nguyen et al. , 2016 ; Wang et al. , 2014 ) , but is limited in the relation systems it can model ( Kazemi & Poole , 2018 ) and does not lead to state of the art performances on current benchmarks . Finally Schlichtkrull et al . ( 2018 ) propose to generate the entity embeddings of a CP-like tensor decomposition by running a forward pass of a Graph Neural Network over the training Knowledge Base . The experiments included in this work did not lead to better link prediction performances than the same decomposition ( Distmult ) directly optimized ( Kadlec et al. , 2017 ) . Temporal link prediction methods Sarkar & Moore ( 2006 ) describes a bayesian model and learning method for representing temporal relations . The temporal smoothness prior used in this work is similar to the gradient penalty we describe in Section 3.3 . However , learning one embedding matrix per timestamp is not applicable to the scales considered in this work . Bader et al . ( 2007 ) uses a tensor decomposition called ASALSAN to express temporal relations . This decomposition is related to RESCAL ( Nickel et al. , 2011 ) which underperforms on recent benchmarks due to overfitting ( Nickel et al. , 2016b ) . For temporal knowledge base completion , Goel et al . ( 2020 ) learns entity embeddings that change over time , by masking a fraction of the embedding weights with an activation function of learned frequencies . Based on the Tucker decomposition , ConT ( Ma et al. , 2018 ) learns one new core tensor for each timestamp . Finally , viewing the time dimension as a sequence to be predicted , GarcíaDurán et al . ( 2018 ) use recurrent neural nets to transform the embeddings of standard models such as TransE or Distmult to accomodate the temporal data . This work follows Lacroix et al . ( 2018 ) by studying and extending a regularized CP decomposition of the training set seen as an order 4 tensor . We propose and study several regularizer suited to our decompositions . 3 MODEL . In this section , we are given facts ( subject , predicate , object ) annotated with timestamps , we discretize the timestamp range ( eg . by reducing timestamps to years ) in order to obtain a training set of 4-tuple ( subject , predicate , object , time ) indexing an order 4 tensor . We will show in Section 5.1 how we reduce each datasets to this setting . Following Lacroix et al . ( 2018 ) , we minimize , for each of the train tuples ( i , j , k , l ) , the instantaneous multiclass loss : ℓ ( X̂ ; ( i , j , k , l ) ) = −X̂i , j , k , l + log ( ∑ k′ exp ( X̂i , j , k′ , l ) ) . ( 1 ) Note that this loss is only suited to queries of the type ( subject , predicate , ? , time ) , which is the queries that were considered in related work . We consider another auxiliary loss in Section 6 which we will use on our Wikidata dataset . For a training set S ( augmented with reciprocal relations ( Lacroix et al. , 2018 ; Kazemi & Poole , 2018 ) ) , and parametric tensor estimate X̂ ( θ ) , we minimize the following objective , with a weighted regularizer Ω : L ( X̂ ( θ ) ) = 1 |S| ∑ ( i , j , k , l ) ∈S [ ℓ ( X̂ ( θ ) ; ( i , j , k , l ) ) + λΩ ( θ ; ( i , j , k , l ) ) ] . The ComplEx ( Trouillon et al. , 2016 ) decomposition can naturally be extended to this setting by adding a new factor T , we then have : X̂ ( U , V , T ) = Re ( [ [ U , V , U , T ] ] ) ⇐⇒ X̂ ( U , V , T ) i , j , k , l = Re ( ⟨ui , vj , uk , tl⟩ ) ( 2 ) We call this decomposition TComplEx . Intuitively , we added timestamps embedding that modulate the multi-linear dot product . Notice that the timestamp can be used to equivalently modulate the objects , predicates or subjects to obtain time-dependent representation : ⟨ui , vj , uk , tl⟩ = ⟨ui ⊙ tl , vj , uk⟩ = ⟨ui , vj ⊙ tl , uk⟩ = ⟨ui , vj , uk ⊙ tl⟩ . Contrary to DE-SimplE ( Goel et al. , 2020 ) , we do not learn temporal embeddings that scale with the number of entities ( as frequencies and biases ) , but rather embeddings that scale with the number of timestamps . The number of parameters for the two models are compared in Table 1 . 3.1 NON-TEMPORAL PREDICATES . Some predicates might not be affected by timestamps . For example , Malia and Sasha will always be the daughters of Barack and Michelle Obama , whereas the “ has occupation ” predicate between two entities might very well change over time . In heterogeneous knowledge bases , where some predicates might be temporal and some might not be , we propose to decompose the tensor X̂ as the sum of two tensors , one temporal , and the other non-temporal : X̂ = Re ( [ [ U , V t , U , T ] ] + [ [ U , V , U,1 ] ] ) ⇐⇒ X̂i , j , k , l = Re ( ⟨ui , vtj ⊙ tl + vj , uk⟩ ) ( 3 ) We call this decomposition TNTComplEx . Goel et al . ( 2020 ) suggests another way of introducing a non-temporal component , by only allowing a fraction γ of components of the embeddings to be modulated in time . By allowing this sharing of parameters between the temporal and non-temporal part of the tensor , our model removes one hyperparameter . Moreover , preliminary experiments showed that this model outperforms one without parameter sharing .

Summary: In this paper, the authors study an important problem, i.e., time-aware link prediction in a knowledge base. Specifically, the authors focus on predicting the missing link in a quadruple, i.e., (subject, predicate, ?, timestamp). In particular, the authors design a new tensor (order 4) factorization based method with proper regularization terms shown in Eqs.(4-6).