Tensor Decompositions for Temporal Knowledge Base Completion

1 INTRODUCTION . Link prediction in relational data has been the subject of interest , given the widespread availability of such data and the breadth of its use in bioinformatics ( Zitnik et al. , 2018 ) , recommender systems ( Koren et al. , 2009 ) or Knowledge Base completion ( Nickel et al. , 2016a ) . Relational data is often temporal , for example , the action of buying an item or watching a movie is associated to a timestamp . Some medicines might not have the same adverse side effects depending on the subject ’ s age . The task of temporal link prediction is to find missing links in graphs at precise points in time . In this work , we study temporal link prediction through the lens of temporal knowledge base completion , which provides varied benchmarks both in terms of the underlying data they represent , but also in terms of scale . A knowledge base is a set of facts ( subject , predicate , object ) about the world that are known to be true . Link prediction in a knowledge base amounts to answer incomplete queries of the form ( subject , predicate , ? ) by providing an accurate ranking of potential objects . In temporal knowledge bases , these facts have some temporal metadata attached . For example , facts might only hold for a certain time interval , in which case they will be annotated as such . Other facts might be event that happened at a certain point in time . Temporal link prediction amounts to answering queries of the form ( subject , predicate , ? , timestamp ) . For example , we expect the ranking of queries ( USA , president , ? , timestamp ) to vary with the timestamps . As tensor factorization methods have proved successful for Knowledge Base Completion ( Nickel et al. , 2016a ; Trouillon et al. , 2016 ; Lacroix et al. , 2018 ) , we express our Temporal Knowledge Base Completion problem as an order 4 tensor completion problem . That is , timestamps are discretized and used to index a 4-th mode in the binary tensor holding ( subject , predicate , object , timestamps ) facts . First , we introduce a ComplEx ( Trouillon et al. , 2016 ) decomposition of this order 4 tensor , and link it with previous work on temporal Knowledge Base completion . This decomposition yields embeddings for each timestamps . A natural prior is for these timestamps representation to evolve slowly over time . We are able to introduce this prior as a regularizer for which the optimum is a ∗Université Paris-Est , Equipe Imagine , LIGM ( UMR8049 ) Ecole des Ponts ParisTech , Marne-la-Vallée variation on the nuclear p-norm . In order to deal with heterogeneous temporal knowledge bases where a significant amount of relations might be non-temporal , we add a non-temporal component to our decomposition . Experiments on available benchmarks show that our method outperforms the state of the art for similar number of parameters . We run additional experiments for larger , regularized models and obtain improvements of up to 0.07 absolute Mean Reciprocal Rank ( MRR ) . Finally , we propose a dataset of 400k entities , based on Wikidata , with 7M train triples , of which 10 % contain temporal validity information . This dataset is larger than usual benchmarks in the Knowledge Base completion community and could help bridge the gap between the method designed and the envisaged web-scale applications . 2 RELATED WORK . Matrices and tensors are upper case letters . The i-th row of U is denoted by ui while it ’ s j − th column is denoted by U : ,j . The tensor product of two vectors is written ⊗ and the hadamard ( elementwise ) product ⊙ . Static link prediction methods Standard tensor decomposition methods have lead to good results ( Yang et al. , 2014 ; Trouillon et al. , 2016 ; Lacroix et al. , 2018 ; Balažević et al. , 2019 ) in Knowledge Base completion . The Canonical Polyadic ( CP ) Decomposition ( Hitchcock , 1927 ) is the tensor equivalent to the low-rank decomposition of a matrix . A tensor X of canonical rank R can be written as : X = R∑ r=1 U : ,r⊗V : ,r⊗W : ,r = [ [ U , V , W ] ] ⇐⇒ ∀ ( i , j , k ) , Xi , j , k = R∑ r=1 ui , rvj , rwk , r = ⟨ui , vj , wk⟩ Setting U = W leads to the Distmult ( Yang et al. , 2014 ) model , which has been successful , despite only being able to represent symmetric score functions . In order to keep the parameter sharing scheme but go beyond symmetric relations , Trouillon et al . ( 2016 ) use complex parameters and set W to the complex conjugate of U , U . Regularizing this algorithm with the variational form of the tensor nuclear norm as well as a slight transformation to the learning objective ( also proposed in Kazemi & Poole ( 2018 ) ) leads to state of the art results in Lacroix et al . ( 2018 ) . Other methods are not directly inspired from classical tensor decompositions . For example , TransE ( Bordes et al. , 2013 ) models the score as a distance of the translated subject to an object representation . This method has lead to many variations ( Ji et al. , 2015 ; Nguyen et al. , 2016 ; Wang et al. , 2014 ) , but is limited in the relation systems it can model ( Kazemi & Poole , 2018 ) and does not lead to state of the art performances on current benchmarks . Finally Schlichtkrull et al . ( 2018 ) propose to generate the entity embeddings of a CP-like tensor decomposition by running a forward pass of a Graph Neural Network over the training Knowledge Base . The experiments included in this work did not lead to better link prediction performances than the same decomposition ( Distmult ) directly optimized ( Kadlec et al. , 2017 ) . Temporal link prediction methods Sarkar & Moore ( 2006 ) describes a bayesian model and learning method for representing temporal relations . The temporal smoothness prior used in this work is similar to the gradient penalty we describe in Section 3.3 . However , learning one embedding matrix per timestamp is not applicable to the scales considered in this work . Bader et al . ( 2007 ) uses a tensor decomposition called ASALSAN to express temporal relations . This decomposition is related to RESCAL ( Nickel et al. , 2011 ) which underperforms on recent benchmarks due to overfitting ( Nickel et al. , 2016b ) . For temporal knowledge base completion , Goel et al . ( 2020 ) learns entity embeddings that change over time , by masking a fraction of the embedding weights with an activation function of learned frequencies . Based on the Tucker decomposition , ConT ( Ma et al. , 2018 ) learns one new core tensor for each timestamp . Finally , viewing the time dimension as a sequence to be predicted , GarcíaDurán et al . ( 2018 ) use recurrent neural nets to transform the embeddings of standard models such as TransE or Distmult to accomodate the temporal data . This work follows Lacroix et al . ( 2018 ) by studying and extending a regularized CP decomposition of the training set seen as an order 4 tensor . We propose and study several regularizer suited to our decompositions . 3 MODEL . In this section , we are given facts ( subject , predicate , object ) annotated with timestamps , we discretize the timestamp range ( eg . by reducing timestamps to years ) in order to obtain a training set of 4-tuple ( subject , predicate , object , time ) indexing an order 4 tensor . We will show in Section 5.1 how we reduce each datasets to this setting . Following Lacroix et al . ( 2018 ) , we minimize , for each of the train tuples ( i , j , k , l ) , the instantaneous multiclass loss : ℓ ( X̂ ; ( i , j , k , l ) ) = −X̂i , j , k , l + log ( ∑ k′ exp ( X̂i , j , k′ , l ) ) . ( 1 ) Note that this loss is only suited to queries of the type ( subject , predicate , ? , time ) , which is the queries that were considered in related work . We consider another auxiliary loss in Section 6 which we will use on our Wikidata dataset . For a training set S ( augmented with reciprocal relations ( Lacroix et al. , 2018 ; Kazemi & Poole , 2018 ) ) , and parametric tensor estimate X̂ ( θ ) , we minimize the following objective , with a weighted regularizer Ω : L ( X̂ ( θ ) ) = 1 |S| ∑ ( i , j , k , l ) ∈S [ ℓ ( X̂ ( θ ) ; ( i , j , k , l ) ) + λΩ ( θ ; ( i , j , k , l ) ) ] . The ComplEx ( Trouillon et al. , 2016 ) decomposition can naturally be extended to this setting by adding a new factor T , we then have : X̂ ( U , V , T ) = Re ( [ [ U , V , U , T ] ] ) ⇐⇒ X̂ ( U , V , T ) i , j , k , l = Re ( ⟨ui , vj , uk , tl⟩ ) ( 2 ) We call this decomposition TComplEx . Intuitively , we added timestamps embedding that modulate the multi-linear dot product . Notice that the timestamp can be used to equivalently modulate the objects , predicates or subjects to obtain time-dependent representation : ⟨ui , vj , uk , tl⟩ = ⟨ui ⊙ tl , vj , uk⟩ = ⟨ui , vj ⊙ tl , uk⟩ = ⟨ui , vj , uk ⊙ tl⟩ . Contrary to DE-SimplE ( Goel et al. , 2020 ) , we do not learn temporal embeddings that scale with the number of entities ( as frequencies and biases ) , but rather embeddings that scale with the number of timestamps . The number of parameters for the two models are compared in Table 1 . 3.1 NON-TEMPORAL PREDICATES . Some predicates might not be affected by timestamps . For example , Malia and Sasha will always be the daughters of Barack and Michelle Obama , whereas the “ has occupation ” predicate between two entities might very well change over time . In heterogeneous knowledge bases , where some predicates might be temporal and some might not be , we propose to decompose the tensor X̂ as the sum of two tensors , one temporal , and the other non-temporal : X̂ = Re ( [ [ U , V t , U , T ] ] + [ [ U , V , U,1 ] ] ) ⇐⇒ X̂i , j , k , l = Re ( ⟨ui , vtj ⊙ tl + vj , uk⟩ ) ( 3 ) We call this decomposition TNTComplEx . Goel et al . ( 2020 ) suggests another way of introducing a non-temporal component , by only allowing a fraction γ of components of the embeddings to be modulated in time . By allowing this sharing of parameters between the temporal and non-temporal part of the tensor , our model removes one hyperparameter . Moreover , preliminary experiments showed that this model outperforms one without parameter sharing .

Summary: This paper extends the ComplEx model (Trouillon et al., 2016) for completing temporal knowledge bases by augmenting it with timestamp embeddings. Besides, based on the assumption that these timestamp representations evolve slowly over time, the paper introduces this prior as a regularizer. Also, the paper adds a non-temporal component to the model to deal with static facts in knowledge bases. The proposed model has been evaluated using the current benchmark temporal event datasets, showing state-of-the-art performance.