Inductive representation learning on temporal graphs

1 INTRODUCTION . The technique of learning lower-dimensional vector embeddings on graphs have been widely applied to graph analysis tasks ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Wang et al. , 2016 ) and deployed in industrial systems ( Ying et al. , 2018 ; Wang et al. , 2018a ) . Most of the graph representation learning approaches only accept static or non-temporal graphs as input , despite the fact that many graph-structured data are time-dependent . In social network , citation network , question answering forum and user-item interaction system , graphs are created as temporal interactions between nodes . Using the final state as a static portrait of the graph is reasonable in some cases , such as the proteinprotein interaction network , as long as node interactions are timeless in nature . Otherwise , ignoring the temporal information can severely diminish the modelling efforts and even causing questionable inference . For instance , models may mistakenly utilize future information for predicting past interactions during training and testing if the temporal constraints are disregarded . More importantly , the dynamic and evolving nature of many graph-related problems demand an explicitly modelling of the timeliness whenever nodes and edges are added , deleted or changed over time . Learning representations on temporal graphs is extremely challenging , and it is not until recently that several solutions are proposed ( Nguyen et al. , 2018 ; Li et al. , 2018 ; Goyal et al. , 2018 ; Trivedi et al. , 2018 ) . We conclude the challenges in three folds . Firstly , to model the temporal dynamics , node embeddings should not be only the projections of topological structures and node features but also functions of the continuous time . Therefore , in addition to the usual vector space , temporal representation learning should be operated in some functional space as well . Secondly , graph topological structures are no longer static since the nodes and edges are evolving over time , which poses ∗Both authors contributed equally to this research . temporal constraints on neighborhood aggregation methods . Thirdly , node features and topological structures can exhibit temporal patterns . For example , node interactions that took place long ago may have less impact on the current topological structure and thus the node embeddings . Also , some nodes may possess features that allows them having more regular or recurrent interactions with others . We provide sketched plots for visual illustration in Figure 1 . Similar to its non-temporal counterparts , in the real-world applications , models for representation learning on temporal graphs should be able to quickly generate embeddings whenever required , in an inductive fashion . GraphSAGE ( Hamilton et al. , 2017a ) and graph attention network ( GAT ) ( Veličković et al. , 2017 ) are capable of inductively generating embeddings for unseen nodes based on their features , however , they do not consider the temporal factors . Most of the temporal graph embedding methods can only handle transductive tasks , since they require re-training or the computationally-expensive gradient calculations to infer embeddings for unseen nodes or node embeddings for a new timepoint . In this work , we aim at developing an architecture to inductively learn representations for temporal graphs such that the time-aware embeddings ( for unseen and observed nodes ) can be obtained via a single network forward pass . The key to our approach is the combination of the self-attention mechanism ( Vaswani et al. , 2017 ) and a novel functional time encoding technique derived from the Bochner ’ s theorem from classical harmonic analysis ( Loomis , 2013 ) . The motivation for adapting self-attention to inductive representation learning on temporal graphs is to identify and capture relevant pieces of the temporal neighborhood information . Both graph convolutional network ( GCN ) ( Kipf & Welling , 2016a ) and GAT are implicitly or explicitly assigning different weights to neighboring nodes ( Veličković et al. , 2017 ) when aggregating node features . The self-attention mechanism was initially designed to recognize the relevant parts of input sequence in natural language processing . As a discrete-event sequence learning method , self-attention outputs a vector representation of the input sequence as a weighted sum of individual entry embeddings . Selfattention enjoys several advantages such as parallelized computation and interpretability ( Vaswani et al. , 2017 ) . Since it captures sequential information only through the positional encoding , temporal features can not be handled . Therefore , we are motivated to replace positional encoding with some vector representation of time . Since time is a continuous variable , the mapping from the time domain to vector space has to be functional . We gain insights from harmonic analysis and propose a theoretical-grounded functional time encoding approach that is compatible with the self-attention mechanism . The temporal signals are then modelled by the interactions between the functional time encoding and nodes features as well as the graph topological structures . To evaluate our approach , we consider future link prediction on the observed nodes as transductive learning task , and on the unseen nodes as inductive learning task . We also examine the dynamic node classification task using node embeddings ( temporal versus non-temporal ) as features to demonstrate the usefulness of our functional time encoding . We carry out extensive ablation studies and sensitivity analysis to show the effectiveness of the proposed functional time encoding and TGAT -layer . 2 RELATED WORK . Graph representation learning . Spectral graph embedding models operate on the graph spectral domain by approximating , projecting or expanding the graph Laplacian ( Kipf & Welling , 2016a ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ) . Since their training and inference are conditioned on the specific graph spectrum , they are not directly extendable to temporal graphs . Non-spectral approaches , such as GAT , GraphSAGE and MoNET , ( Monti et al. , 2017 ) rely on the localized neighbourhood aggregations and thus are not restricted to the training graph . GraphSAGE and GAT also have the flexibility to handle evolving graphs inductively . To extend classical graph representation learning approaches to the temporal domain , several attempts have been done by cropping the temporal graph into a sequence of graph snapshots ( Li et al. , 2018 ; Goyal et al. , 2018 ; Rahman et al. , 2018 ; Xu et al. , 2019b ) , and some others work with temporally persistent node ( edges ) ( Trivedi et al. , 2018 ; Ma et al. , 2018 ) . Nguyen et al . ( 2018 ) proposes a node embedding method based on temporal random walk and reported state-of-the-art performances . However , their approach only generates embeddings for the final state of temporal graph and can not directly apply to the inductive setting . Self-attention mechanism . Self-attention mechanisms often have two components : the embedding layer and the attention layer . The embedding layer takes an ordered entity sequence as input . Selfattention uses the positional encoding , i.e . each position k is equipped with a vector pk ( fixed or learnt ) which is shared for all sequences . For the entity sequence e = ( e1 , . . . , el ) , the embedding layer takes the sum or concatenation of entity embeddings ( or features ) ( z ∈ Rd ) and their positional encodings as input : Ze = [ ze1 + p1 , . . . , ze1 + pl ] ᵀ ∈ Rl×d , or Ze = [ ze1 ||p1 , . . . , ze1 ||pl ] ᵀ ∈ Rl× ( d+dpos ) . ( 1 ) where || denotes concatenation operation and dpos is the dimension for positional encoding . Selfattention layers can be constructed using the scaled dot-product attention , which is defined as : Attn ( Q , K , V ) = softmax ( QKᵀ√ d ) V , ( 2 ) where Q denotes the ’ queries ’ , K the ’ keys ’ and V the ’ values ’ . In Vaswani et al . ( 2017 ) , they are treated as projections of the output Ze : Q = ZeWQ , K = ZeWK , V = ZeWV , where WQ , WK and WV are the projection matrices . Since each row of Q , K and V represents an entity , the dot-product attention takes a weighted sum of the entity ’ values ’ in V where the weights are given by the interactions of entity ’ query-key ’ pairs . The hidden representation for the entity sequence under the dot-product attention is then given by he = Attn ( Q , K , V ) . 3 TEMPORAL GRAPH ATTENTION NETWORK ARCHITECTURE . We first derive the mapping from time domain to the continuous differentiable functional domain as the functional time encoding such that resulting formulation is compatible with self-attention mechanism as well as the backpropagation-based optimization frameworks . The same idea was explored in a concurrent work ( Xu et al. , 2019a ) . We then present the temporal graph attention layer and show how it can be naturally extended to incorporate the edge features . 3.1 FUNCTIONAL TIME ENCODING . Recall that our starting point is to obtain a continuous functional mapping Φ : T → RdT from time domain to the dT -dimensional vector space to replace the positional encoding in ( 1 ) . Without loss of generality , we assume that the time domain can be represented by the interval starting from origin : T = [ 0 , tmax ] , where tmax is determined by the observed data . For the inner-product selfattention in ( 2 ) , often the ’ key ’ and ’ query ’ matrices ( K , Q ) are given by identity or linear projection of Ze defined in ( 1 ) , leading to terms that only involve inner-products between positional ( time ) encodings . Consider two time points t1 , t2 and inner product between their functional encodings〈 Φ ( t1 ) , Φ ( t2 ) 〉 . Usually , the relative timespan , rather than the absolute value of time , reveals critical temporal information . Therefore , we are more interested in learning patterns related to the timespan of |t2−t1| , which should be ideally expressed by 〈 Φ ( t1 ) , Φ ( t2 ) 〉 to be compatible with self-attention . Formally , we define the temporal kernel K : T × T → R with K ( t1 , t2 ) : = 〈 Φ ( t1 ) , Φ ( t2 ) 〉 and K ( t1 , t2 ) = ψ ( t1 − t2 ) , ∀t1 , t2 ∈ T for some ψ : [ −tmax , tmax ] → R. The temporal kernel is then translation-invariant , since K ( t1 + c , t2 + c ) = ψ ( t1 − t2 ) = K ( t1 , t2 ) for any constant c. Generally speaking , functional learning is extremely complicated since it operates on infinite-dimensional spaces , but now we have transformed the problem into learning the temporal kernel K expressed by Φ . Nonetheless , we still need to figure out an explicit parameterization for Φ in order to conduct efficient gradient-based optimization . Classical harmonic analysis theory , i.e . the Bochner ’ s theorem , motivates our final solution . We point out that the temporal kernel K is positive-semidefinite ( PSD ) and continuous , since it is defined via Gram matrix and the mapping Φ is continuous . Therefore , the kernel K defined above satisfy the assumptions of the Bochner ’ s theorem , which we state below . Theorem 1 ( Bochner ’ s Theorem ) . A continuous , translation-invariant kernel K ( x , y ) = ψ ( x− y ) on Rd is positive definite if and only if there exists a non-negative measure on R such that ψ is the Fourier transform of the measure . Consequently , when scaled properly , our temporal kernel K have the alternate expression : K ( t1 , t2 ) = ψ ( t1 , t2 ) = ∫ R eiω ( t1−t2 ) p ( ω ) dω = Eω [ ξω ( t1 ) ξω ( t2 ) ∗ ] , ( 3 ) where ξω ( t ) = eiωt . Since the kernel K and the probability measure p ( ω ) are real , we extract the real part of ( 3 ) and obtain : K ( t1 , t2 ) = Eω [ cos ( ω ( t1 − t2 ) ) ] = Eω [ cos ( ωt1 ) cos ( ωt2 ) + sin ( ωt1 ) sin ( ωt2 ) ] . ( 4 ) The above formulation suggests approximating the expectation by the Monte Carlo integral ( Rahimi & Recht , 2008 ) , i.e . K ( t1 , t2 ) ≈ 1d ∑d i=1 cos ( ωit1 ) cos ( ωit2 ) + sin ( ωit1 ) sin ( ωit2 ) , with ω1 , . . . , ωd i.i.d∼ p ( ω ) . Therefore , we propose the finite dimensional functional mapping to Rd as : t 7→ Φd ( t ) : = √ 1 d [ cos ( ω1t ) , sin ( ω1t ) , . . . , cos ( ωdt ) , sin ( ωdt ) ] , ( 5 ) and it is easy to show that 〈 Φd ( t1 ) , Φd ( t2 ) 〉 ≈ K ( t1 , t2 ) . As a matter of fact , we prove the stochastic uniform convergence of 〈 Φd ( t1 ) , Φd ( t2 ) 〉 to the underlying K ( t1 , t2 ) and shows that it takes only a reasonable amount of samples to achieve proper estimation , which is stated in Claim 1 . Claim 1 . Let p ( ω ) be the corresponding probability measure stated in Bochner ’ s Theorem for kernel function K. Suppose the feature map Φ is constructed as described above using samples { ωi } di=1 , then we only need d = Ω ( 1 2 log σ2ptmax ) samples to have sup t1 , t2∈T ∣∣Φd ( t1 ) ′Φd ( t2 ) −K ( t1 , t2 ) ∣∣ < with any probability for ∀ > 0 , where σ2p is the second momentum with respect to p ( ω ) . The proof is provided in supplement material . By applying Bochner ’ s theorem , we convert the problem of kernel learning to distribution learning , i.e . estimating the p ( ω ) in Theorem 1 . A straightforward solution is to apply the reparameterization trick by using auxiliary random variables with a known marginal distribution as in variational autoencoders ( Kingma & Welling , 2013 ) . However , the reparameterization trick is often limited to certain distributions such as the ’ local-scale ’ family , which may not be rich enough for our purpose . For instance , when p ( ω ) is multimodal it is difficult to reconstruct the underlying distribution via direct reparameterizations . An alternate approach is to use the inverse cumulative distribution function ( CDF ) transformation . Rezende & Mohamed ( 2015 ) propose using parameterized normalizing flow , i.e . a sequence of invertible transformation functions , to approximate arbitrarily complicated CDF and efficiently sample from it . Dinh et al . ( 2016 ) further considers stacking bijective transformations , known as affine coupling layer , to achieve more effective CDF estimation . The above methods learns the inverse CDF function F−1θ ( . ) parameterized by flow-based networks and draw samples from the corresponding distribution . On the other hand , if we consider an non-parameterized approach for estimating distribution , then learning F−1 ( . ) and obtain d samples from it is equivalent to directly optimizing the { ω1 , . . . , ωd } in ( 4 ) as free model parameters . In practice , we find these two approaches to have highly comparable performances ( see supplement material ) . Therefore we focus on the non-parametric approach , since it is more parameter-efficient and has faster training speed ( as no sampling during training is required ) . The above functional time encoding is fully compatible with self-attention , thus they can replace the positional encodings in ( 1 ) and their parameters are jointly optimized as part of the whole model .

Summary: This paper proposed the temporal graph attention layer which aggregates in-hop features with self-attention and incorporates temporal information with Fourier based relative positional encoding. This idea is novel in GCN field. Experimental results demonstrate that the TGAT which adds temporal encoding outperforms the other methods. Overall this paper addressed its core ideas clearly and made proper experiments and analysis to demonstrate the superiority against existing counterparts.