TaCE: Time-aware Convolutional Embedding Learning for Temporal Knowledge Graph Completion

1 INTRODUCTION . Knowledge graphs ( KGs ) , storing facts in tuples , are often faced with incompletion . To solve this problem , knowledge graph embedding ( KGE ) , mapping the entities and relations into a continuous vector space , has been developed to capture the semantic meanings for the task of KG reasoning ( KGR ) or KG completion ( KGC ) ( Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Sun et al. , 2019 ) . Both KGR and KGC aim at inferring the missing facts for a given knowledge graph ( Chen et al. , 2020 ) . Traditional KGE or KGC approaches usually treat KGs to be static , which means that the nodes and edges of a KG would not evolve with the time ( Kazemi et al. , 2020 ; Ji et al. , 2021 ) . However , in reality , most facts or events are only valid at a specific point or over a certain period . For example , in “ Franklin D.Roosevelt was in office during 1933-1945 and died on April 12th , 1945 ” , it indicates two facts ( ‘ in office ’ and ‘ death ’ ) related to the target person following different time orders and spans . In that case , KGC should be implemented at the temporal scale , and KGE models are supposed to have temporal awareness . To better capture the knowledge evolution , recent temporal KG reasoning ( TKGR ) or temporal KG completion ( TKGC ) researches try to integrate the temporal information into the KGE procedure . Methods can be roughly divided into two categories : the structure-based methods and the sequencebased methods . Acting as an extension of static KGE modelling , the former type aims to project the entities and relations into a time-dependent vector space with the inherent KG structure preserved ( Jiang et al. , 2016 ; Dasgupta et al. , 2018 ; Xu et al. , 2020 ; Lacroix et al. , 2020 ) ; they behave to be time-efficient on account of linear or multi-linear transformations but face with the shallow representation problem . The sequence-based methods ( Jin et al. , 2020 ; Wu et al. , 2020 ; Li et al. , 2021 ) , splitting the entire KG into a sequence of graph snapshots along the time , rely on the sequence models such as the recurrent neural network ( RNN ) ( Jin et al. , 2020 ; Trivedi et al. , 2017 ; Seo et al. , 2018 ) , long and short-term memory ( LSTM ) ( Wu et al. , 2020 ) or gated recurrent unit ( GRU ) ( Li et al. , 2021 ) to inherently encode temporal features . Although the performances based on the sequence models are reasonably good , they are computationally expensive when running over a large-scale KG . Moreover , their prediction results would be undermined by temporal sparsity to some extent due to the reason that only a small fraction of nodes and edges are activated at each time ( Wu et al. , 2020 ) . In this paper , we are inclined to apply the structure-based method to manage TKGR or TKGC . Inspired by the successful applications of convolutional networks on static KGE ( Dettmers et al. , 2018 ; Jiang et al. , 2019 ; Balažević et al. , 2019a ) , we hereby build a novel time-aware convolutional embedding ( TaCE ) model to infer the missing facts for a temporal knowledge graph ( TKG ) . It highlights its advantage in generating the convolution filters constructed from timestamps and convolving the time information into the embeddings of entities and relations . Such a convolutional design enables the model to deeply , comprehensively and efficiently extract the temporal features and the static ones and achieves a better link prediction for the TKGC task . Results demonstrate that TaCE has the superiority in learning temporal information as well as its interactions with the entities and the relations . It balances the tradeoff between expressiveness and training speed . The key contributions of this article can be summarized as follows : • We creatively propose time-aware convolution method to integrate the time information into the embeddings of entities and relations for a better link prediction . • An extensive comparison has been conducted between TaCE and the state-of-the-art models to verify their performances on different public datasets . • Further analyses have been done to prove that our model , TaCE , is able to capture the semantic meanings for the timestamps and learn facts with proper time order and consistency . 2 RELATED WORK . Static KG representation learning To discover the unknown facts in KGs , substantial static KGE methods have been proposed in the last decade . These methods commonly convert entities and relations into continuous vector spaces , and employ a scoring function to measure the plausibility of each candidate for KGC . TransE ( Bordes et al. , 2013 ) , is one of the most widely-used transitional distance models , to embed entities and relations . Motivated by TransE , a series of similar models including TransH ( Wang et al. , 2014 ) , TransR ( Lin et al. , 2015 ) , and TransD ( Ji et al. , 2015 ) are developed to achieve better link predictions . RESCAL ( Nickel et al. , 2011 ) and its extensions ( DistMult ( Yang et al. , 2014 ) , ComplEx ( Trouillon et al. , 2016 ) , and TuckER ( Balažević et al. , 2019b ) ) represent typical semantic matching models using tensor/matric factorization . Apart from them , ConvE ( Dettmers et al. , 2018 ) , ConvR ( Jiang et al. , 2019 ) and HypER ( Balažević et al. , 2019a ) recently ignite the passions of using convolution networks for KGE ; they successfully prove their feature expressiveness better than those linear ones in link prediction . However , there is no study by far introducing convolution modelling into TKGC . Temporal KG representation learning As forementioned , representation learning for TKG can be roughly categorized into two classes : structure-based models and sequence-based models . The former represent the quadruples in TKG by building time-sensitive embedding models . TTransE ( Jiang et al. , 2016 ) and DE-SimplE ( Goel et al. , 2020 ) integrate timestamp information into the corresponding the embeddings of relations and entities to infer the missing knowledge . HyTE ( Dasgupta et al. , 2018 ) projects the entities and relations to the temporal hyperplanes . ChronoR ( Sadeghian et al. , 2021 ) and TeRo ( Xu et al. , 2020 ) , the most recent works , treat the timestamped relations as a temporal rotation from the head entity to the tail entity . Sequence-based models , including KnowEvolve ( Trivedi et al. , 2017 ) , GCRN ( Seo et al. , 2018 ) , RE-NET ( Jin et al. , 2020 ) and RE-GCN ( Li et al. , 2021 ) , attempt to use the sequential networks such as RNN , LSTM and GRU to learn timedependent facts . However , they are designed for the task of graph extrapolation and “ not compatible with TKGC settings ( Wu et al. , 2020 ) . TeMP ( Wu et al. , 2020 ) is one of the sequence-based models , designed for TKGC tasks using graph neural network ( GNN ) and LSTM to capture the intra-graph patterns and inter-graph relationships , respectively . Therefore , we group TeMP , as the representative for the sequence-based model , into our baselines . 3 PROBLEM FORMULATION . Before going to the details of our TaCE model , we formally define the key notations and TKGC task . Temporal Knowledge Graph ( TKG ) A TKG G is composed by a set of real-world facts G = { ( s , r , o , t ) | s , r ∈ E , r ∈ R , t ∈ T } , where , in each quadruple , s denotes the head entity , r is the relation , o is the tail entity , t is a discretized timestamp ; E , R and T stand for the all the entities , relations and timestamps belong to the TKG . Temporal Knowledge Graph Completion ( TKGC ) The task of TKGC or TKGR is to infer the missing component , ( ? , r , o , t ) , ( s , r , ? , t ) or ( s , ? , o , t ) , given that the other three elements of the quadruple are known . It is supposed that the missing part exists in E , R. In this article , our job focuses on predicting the missing head or tail entity , but we only take ( s , r , ? , t ) as the example for modelling expression in later section . 4 METHOD : TACE . 4.1 FRAMEWORK OF TACE . The model of TaCE is mainly composed of four components : the temporal convolution module aims to construct temporal convolution filters to go through the input entities and relations to obtain the time-aware representations for all the input ; the static convolution module , as the name suggests , is to facilitate the convolution across all the entities and relations again to learn the context information changed without the time ; the deep learning module , made up of several hidden layers , is responsible for comprehensively and deeply drawing the representations carried both from the temporal and static modules ; after that , the prediction layer delivers the probability for each tail entity candidate to suggest the most likely answer for the incomplete G. Figure 1 displays the architecture of TaCE . 4.2 TEMPORAL CONVOLUTION MODULE . In this module , the temporal information formed by the timestamp t ∈ T is fully integrated into the subject entity s ∈ E and the relation r ∈ R via the convolution filters adaptively constructed from the timestamp t. • Firstly , to integrate the time information into the entity s and the relation r , the temporal convolution filters are developed . They are constructed from the timestamp embedding et ∈ Rdt , where dt is the embedding size of the timestamps . Originally derived from the timestamp t , et is the input into a fully connected layer fc to get a vector vt with a proper length for further processing ; then et is further split into a set of 1D convolution filters Ft = { k ( 1 ) t , k ( 2 ) t , . . . , k ( c ) t } sharing the same size , where k ( l ) t ∈ Rw represents the lth convolution filter , w denotes the embedding size of the filters and c denotes the number of the filters . The procedure of constructing temporal convolution filters is illustrated in Figure 2 . • Secondly , before filtered by temporal convolutional filters Ft , the subject entity embedding es ∈ Rde and the relational embedding er ∈ Rdr are stacked up like an ‘ image ’ with multi channels , with de = dr arepresenting the embedding size of entities and relations respectively . Such a stacking operation enables both the entity information and relation information to feasibly act with Ft. After convolved by Ft , the corresponding temporal feature maps Mtemp = { m ( 1 ) t , m ( 2 ) t , . . . , m ( c ) t } , where m ( l ) t ∈ Rde−w+1 is the lth feature map convolved from the subject entity embedding es and the relational embedding er , and c is equal to the number of filters . The convolution with temporal convolution filters is illustrated in Figure 3. hs and hr denote the convolved es and er respectively in Figure 3 . • Finally , the matrix Mtemp feed into a fully connected layer ftemp to obtain the flatten knowledge feature of atemp ∈ Rde . The equation of the temporal block is formulated as follow : atemp = ftemp ( [ es ; er ] 3 ∗ vec −1 ( fc ( et ) ) ) ( 1 ) where , vec−1 is a splitting operator to reshape the embedding et of the timestamp t into a set of filters , ∗ represents the convolutional operations , and [ es ; er ] 3 represents the stacked tensor made up of the embedding es and the embedding er .

Summary: The paper proposes a novel and more expressive model TaCE for temporal knowledge base completion. TaCE has 4 modules - the Temporal Convolution module (generates a time-aware representation of (s,r) or (r,o) depending on the query), Static Convolution module (generates a time-independent representation of (s,r) or (r,o) depending on the query), Deep Learning module (output of previous two modules is passed through various fully-connected layers), and Link prediction module (this final module takes input from deep learning module and scores all entities on their likelihood of being the correct answer to the query). The superiority of the model is demonstrated via extensive experimentation.