TaCE: Time-aware Convolutional Embedding Learning for Temporal Knowledge Graph Completion

1 INTRODUCTION . Knowledge graphs ( KGs ) , storing facts in tuples , are often faced with incompletion . To solve this problem , knowledge graph embedding ( KGE ) , mapping the entities and relations into a continuous vector space , has been developed to capture the semantic meanings for the task of KG reasoning ( KGR ) or KG completion ( KGC ) ( Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Sun et al. , 2019 ) . Both KGR and KGC aim at inferring the missing facts for a given knowledge graph ( Chen et al. , 2020 ) . Traditional KGE or KGC approaches usually treat KGs to be static , which means that the nodes and edges of a KG would not evolve with the time ( Kazemi et al. , 2020 ; Ji et al. , 2021 ) . However , in reality , most facts or events are only valid at a specific point or over a certain period . For example , in “ Franklin D.Roosevelt was in office during 1933-1945 and died on April 12th , 1945 ” , it indicates two facts ( ‘ in office ’ and ‘ death ’ ) related to the target person following different time orders and spans . In that case , KGC should be implemented at the temporal scale , and KGE models are supposed to have temporal awareness . To better capture the knowledge evolution , recent temporal KG reasoning ( TKGR ) or temporal KG completion ( TKGC ) researches try to integrate the temporal information into the KGE procedure . Methods can be roughly divided into two categories : the structure-based methods and the sequencebased methods . Acting as an extension of static KGE modelling , the former type aims to project the entities and relations into a time-dependent vector space with the inherent KG structure preserved ( Jiang et al. , 2016 ; Dasgupta et al. , 2018 ; Xu et al. , 2020 ; Lacroix et al. , 2020 ) ; they behave to be time-efficient on account of linear or multi-linear transformations but face with the shallow representation problem . The sequence-based methods ( Jin et al. , 2020 ; Wu et al. , 2020 ; Li et al. , 2021 ) , splitting the entire KG into a sequence of graph snapshots along the time , rely on the sequence models such as the recurrent neural network ( RNN ) ( Jin et al. , 2020 ; Trivedi et al. , 2017 ; Seo et al. , 2018 ) , long and short-term memory ( LSTM ) ( Wu et al. , 2020 ) or gated recurrent unit ( GRU ) ( Li et al. , 2021 ) to inherently encode temporal features . Although the performances based on the sequence models are reasonably good , they are computationally expensive when running over a large-scale KG . Moreover , their prediction results would be undermined by temporal sparsity to some extent due to the reason that only a small fraction of nodes and edges are activated at each time ( Wu et al. , 2020 ) . In this paper , we are inclined to apply the structure-based method to manage TKGR or TKGC . Inspired by the successful applications of convolutional networks on static KGE ( Dettmers et al. , 2018 ; Jiang et al. , 2019 ; Balažević et al. , 2019a ) , we hereby build a novel time-aware convolutional embedding ( TaCE ) model to infer the missing facts for a temporal knowledge graph ( TKG ) . It highlights its advantage in generating the convolution filters constructed from timestamps and convolving the time information into the embeddings of entities and relations . Such a convolutional design enables the model to deeply , comprehensively and efficiently extract the temporal features and the static ones and achieves a better link prediction for the TKGC task . Results demonstrate that TaCE has the superiority in learning temporal information as well as its interactions with the entities and the relations . It balances the tradeoff between expressiveness and training speed . The key contributions of this article can be summarized as follows : • We creatively propose time-aware convolution method to integrate the time information into the embeddings of entities and relations for a better link prediction . • An extensive comparison has been conducted between TaCE and the state-of-the-art models to verify their performances on different public datasets . • Further analyses have been done to prove that our model , TaCE , is able to capture the semantic meanings for the timestamps and learn facts with proper time order and consistency . 2 RELATED WORK . Static KG representation learning To discover the unknown facts in KGs , substantial static KGE methods have been proposed in the last decade . These methods commonly convert entities and relations into continuous vector spaces , and employ a scoring function to measure the plausibility of each candidate for KGC . TransE ( Bordes et al. , 2013 ) , is one of the most widely-used transitional distance models , to embed entities and relations . Motivated by TransE , a series of similar models including TransH ( Wang et al. , 2014 ) , TransR ( Lin et al. , 2015 ) , and TransD ( Ji et al. , 2015 ) are developed to achieve better link predictions . RESCAL ( Nickel et al. , 2011 ) and its extensions ( DistMult ( Yang et al. , 2014 ) , ComplEx ( Trouillon et al. , 2016 ) , and TuckER ( Balažević et al. , 2019b ) ) represent typical semantic matching models using tensor/matric factorization . Apart from them , ConvE ( Dettmers et al. , 2018 ) , ConvR ( Jiang et al. , 2019 ) and HypER ( Balažević et al. , 2019a ) recently ignite the passions of using convolution networks for KGE ; they successfully prove their feature expressiveness better than those linear ones in link prediction . However , there is no study by far introducing convolution modelling into TKGC . Temporal KG representation learning As forementioned , representation learning for TKG can be roughly categorized into two classes : structure-based models and sequence-based models . The former represent the quadruples in TKG by building time-sensitive embedding models . TTransE ( Jiang et al. , 2016 ) and DE-SimplE ( Goel et al. , 2020 ) integrate timestamp information into the corresponding the embeddings of relations and entities to infer the missing knowledge . HyTE ( Dasgupta et al. , 2018 ) projects the entities and relations to the temporal hyperplanes . ChronoR ( Sadeghian et al. , 2021 ) and TeRo ( Xu et al. , 2020 ) , the most recent works , treat the timestamped relations as a temporal rotation from the head entity to the tail entity . Sequence-based models , including KnowEvolve ( Trivedi et al. , 2017 ) , GCRN ( Seo et al. , 2018 ) , RE-NET ( Jin et al. , 2020 ) and RE-GCN ( Li et al. , 2021 ) , attempt to use the sequential networks such as RNN , LSTM and GRU to learn timedependent facts . However , they are designed for the task of graph extrapolation and “ not compatible with TKGC settings ( Wu et al. , 2020 ) . TeMP ( Wu et al. , 2020 ) is one of the sequence-based models , designed for TKGC tasks using graph neural network ( GNN ) and LSTM to capture the intra-graph patterns and inter-graph relationships , respectively . Therefore , we group TeMP , as the representative for the sequence-based model , into our baselines . 3 PROBLEM FORMULATION . Before going to the details of our TaCE model , we formally define the key notations and TKGC task . Temporal Knowledge Graph ( TKG ) A TKG G is composed by a set of real-world facts G = { ( s , r , o , t ) | s , r ∈ E , r ∈ R , t ∈ T } , where , in each quadruple , s denotes the head entity , r is the relation , o is the tail entity , t is a discretized timestamp ; E , R and T stand for the all the entities , relations and timestamps belong to the TKG . Temporal Knowledge Graph Completion ( TKGC ) The task of TKGC or TKGR is to infer the missing component , ( ? , r , o , t ) , ( s , r , ? , t ) or ( s , ? , o , t ) , given that the other three elements of the quadruple are known . It is supposed that the missing part exists in E , R. In this article , our job focuses on predicting the missing head or tail entity , but we only take ( s , r , ? , t ) as the example for modelling expression in later section . 4 METHOD : TACE . 4.1 FRAMEWORK OF TACE . The model of TaCE is mainly composed of four components : the temporal convolution module aims to construct temporal convolution filters to go through the input entities and relations to obtain the time-aware representations for all the input ; the static convolution module , as the name suggests , is to facilitate the convolution across all the entities and relations again to learn the context information changed without the time ; the deep learning module , made up of several hidden layers , is responsible for comprehensively and deeply drawing the representations carried both from the temporal and static modules ; after that , the prediction layer delivers the probability for each tail entity candidate to suggest the most likely answer for the incomplete G. Figure 1 displays the architecture of TaCE . 4.2 TEMPORAL CONVOLUTION MODULE . In this module , the temporal information formed by the timestamp t ∈ T is fully integrated into the subject entity s ∈ E and the relation r ∈ R via the convolution filters adaptively constructed from the timestamp t. • Firstly , to integrate the time information into the entity s and the relation r , the temporal convolution filters are developed . They are constructed from the timestamp embedding et ∈ Rdt , where dt is the embedding size of the timestamps . Originally derived from the timestamp t , et is the input into a fully connected layer fc to get a vector vt with a proper length for further processing ; then et is further split into a set of 1D convolution filters Ft = { k ( 1 ) t , k ( 2 ) t , . . . , k ( c ) t } sharing the same size , where k ( l ) t ∈ Rw represents the lth convolution filter , w denotes the embedding size of the filters and c denotes the number of the filters . The procedure of constructing temporal convolution filters is illustrated in Figure 2 . • Secondly , before filtered by temporal convolutional filters Ft , the subject entity embedding es ∈ Rde and the relational embedding er ∈ Rdr are stacked up like an ‘ image ’ with multi channels , with de = dr arepresenting the embedding size of entities and relations respectively . Such a stacking operation enables both the entity information and relation information to feasibly act with Ft. After convolved by Ft , the corresponding temporal feature maps Mtemp = { m ( 1 ) t , m ( 2 ) t , . . . , m ( c ) t } , where m ( l ) t ∈ Rde−w+1 is the lth feature map convolved from the subject entity embedding es and the relational embedding er , and c is equal to the number of filters . The convolution with temporal convolution filters is illustrated in Figure 3. hs and hr denote the convolved es and er respectively in Figure 3 . • Finally , the matrix Mtemp feed into a fully connected layer ftemp to obtain the flatten knowledge feature of atemp ∈ Rde . The equation of the temporal block is formulated as follow : atemp = ftemp ( [ es ; er ] 3 ∗ vec −1 ( fc ( et ) ) ) ( 1 ) where , vec−1 is a splitting operator to reshape the embedding et of the timestamp t into a set of filters , ∗ represents the convolutional operations , and [ es ; er ] 3 represents the stacked tensor made up of the embedding es and the embedding er .

Summary: The authors propose a novel temporal knowledge graph embedding approach that is based on convolutional neural networks. To create a feature on which the link prediction is performed, the presented approach first creates both a static and a temporal feature (both in their own respective convolution modules) and fuses them together in a third deep learning module. The so generated general feature is then used to calculate scores for the missing element in a tuple (s,r,o,t). The experimental section provides a performance overview of the proposed TaCE approach on standard benchmark datasets for TKGC tasks (ICEWS{14|05-15} and GDELD). As seen in the evaluation, the proposed approach performs better or on-par with most recent SOTA approaches in the field. The result presentation is followed by a more detailed analysis of the learned temporal embeddings that appear to contain a good semantic representation of temporal order. An ablation study further shows the huge impact that the temporal embedding module has on the overall results.