Dynamic Graph Representation Learning via Graph Transformer Networks

1 INTRODUCTION . In recent years , graph representation learning has been recognized as a fundamental learning problem and has received much attention due to its widespread use in various domains , including social network analysis ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) , traffic prediction ( Cui et al. , 2019 ; Rahimi et al. , 2018 ) , knowledge graphs ( Wang et al. , 2019a ; b ) , drug discovery ( Do et al. , 2019 ; Duvenaud et al. , 2015 ) , and recommendation systems ( Berg et al. , 2017 ; Ying et al. , 2018 ) . Most existing graph representation learning work focuses on static graphs . However , real-world graphs are intrinsically dynamic where nodes and edges can appear and disappear over time . This dynamic nature of real-world graphs motivates dynamic graph representation learning methods that can model temporal evolutionary patterns and accurately predict node properties and future edges . Recently , several attempts ( Sankar et al. , 2018 ; Pareja et al. , 2020 ; Goyal et al. , 2018 ) have been made to generalize graph learning algorithms from static graphs to dynamic graphs by first learning node representations on each static graph snapshot then aggregating these representations from the temporal dimension . However , these methods are vulnerable to noisy information such as missing or spurious links . This is due to the ineffective message aggregation over unrelated neighbors from noisy connections . The temporal aggregation makes this issue severe by further carrying the noise information over time . Over-relying on graph structures makes the model sensitive to noisy input and can significantly affect downstream task accuracy . A remedy is to consider the input graph as fully connected and learn a graph topology by assigning lower weights to task-irrelevant edges during training ( Devlin et al. , 2019 ) . However , completely ignoring the graph structure makes the optimization inefficient because the model has to estimate the underlying graph structure while learn model parameters at the same time . To resolve the above challenges , we propose a Transformer-based dynamic graph learning method named Dynamic Graph Transformer ( DGT ) that can “ leverage underlying graph structures ” and “ capture implicit edge connections ” to balance this trade-off . Transformers ( Vaswani et al. , 2017 ) , designed to automatically capture the inter-dependencies between tokens in a sequence , have been successfully applied in several domains such as Natural Language Processing ( Devlin et al. , 2019 ; Brown et al. , 2020 ) and Computer Vision ( Dosovitskiy et al. , 2020 ; Liu et al. , 2021 ) . We summarize the success of Transformers into three main factors , which can also help resolve the aforementioned challenges in dynamic graph representation learning : ( 1 ) fully-connected self-attention : by modeling all pair-wise node relations , DGT can capture implicit edge connections , thus become robust to graphs with noisy information such as missing links ; ( 2 ) positional encoding : by generalizing positional encoding to the graph domain using spatial-temporal encoding , we can inject both spatial and temporal graph evolutionary information as inductive biases into DGT to learn a graph ’ s evolutionary patterns over time ; ( 3 ) self-supervised pre-training : by optimizing two complementary pre-training tasks , DGT presents a better performance on the downstream tasks . Though powerful , training Transformers on large-scale graphs is non-trivial due to the quadratic complexity of the fully connected self-attention on the graph size ( Zaheer et al. , 2020 ; Wang et al. , 2020 ) . This issue is more severe on dynamic graphs as the computation cost grows with the number of time-steps in a dynamic graph ( Pareja et al. , 2020 ; Sankar et al. , 2018 ) . To make the training scalable and independent of both the graph size and the number of time-steps , we first propose a temporal-union graph structure that aggregates graph information from multiple time-steps into a unified meta-graph ; we then develop a two-tower architecture with a novel target-context node sampling strategy to model a subset of nodes with their contextual information . These approaches improve DGT ’ s training efficiency and scalability from both the temporal and spatial perspectives . To this end , we summarize our contributions as follows : ( 1 ) a two-tower Transformer-based method named DGT with spatial-temporal encoding that can capture implicit edge connections in addition to the input graph topology ; ( 2 ) a temporal-union graph data structure that efficiently summarizes the spatial-temporal information of dynamic graphs and a novel target-context node sampling strategy for large-scale training ; ( 3 ) two complementary pre-training tasks that can facilitate performing downstream tasks and are proven beneficial using information theory ; and ( 4 ) a comprehensive evaluation on real-world datasets with ablation studies to validate the effectiveness of DGT . 2 PRELIMINARIES AND RELATED WORKS . In this section , we first define dynamic graphs , then review related literature on dynamic graph representation learning and Transformers on graphs . Dynamic graph definition . The nodes and edges in a dynamic graph may appear and disappear over time . In this paper , we define a dynamic graph as a sequence of static graph snapshots with a temporal order G , { G1 , . . . , GT } , where the t-th snapshot graph Gt ( V , Et ) is an undirected graph with a shared node set V of all time steps and an edge set Et . We also denote its adjacency matrix as At . Our goal is to learn a latent representation of each node at each time-step t , such that the learned representation can be used for any specific downstream task such as link prediction or node classification . Please notice that the shared node set V is not static and will be updated when new snapshot graph arrives , which is the same as Sankar et al . ( 2018 ) ; Pareja et al . ( 2020 ) . Dynamic graph representation learning . Previous dynamic graph representation learning methods usually extend static graph algorithms by further taking the temporal information into consideration . They can mainly be classified into three categories : ( 1 ) smoothness-based methods learn a graph autoencoder to generate node embeddings on each graph snapshot and ensure the temporal smoothness of the node embeddings across consecutive time-steps . For example , DYGEM ( Goyal et al. , 2018 ) uses the learned embeddings from the previous time-step to initialize the embeddings in the next time-step . DYNAERNN applies RNN to smooth node embeddings at different timesteps ; ( 2 ) Recurrent-based methods capture the temporal dependency using RNN . For example , GCRN ( Seo et al. , 2018 ) first computes node embeddings on each snapshot using GCN ( Defferrard et al. , 2016 ) , then feeds the node embeddings into an RNN to learn their temporal dependency . EVOLVEGCN ( Pareja et al. , 2020 ) uses RNN to estimate the GCN weight parameters at different time-steps ; ( 3 ) Attention-based methods use self-attention mechanism for both spatial and temporal message aggregation . For example , DYSAT ( Sankar et al. , 2018 ) propose to use the self-attention mechanism for temporal and spatial information aggregation . TGAT ( Xu et al. , 2020 ) encodes the temporal information into the node feature , then applies self-attention on the temporal augmented node features . However , smoothness-based methods heavily rely on the temporal smoothness and are inadequate when nodes exhibit vastly different evolutionary behaviors , recurrent-based methods scale poorly when the number of time-steps increases due to the recurrent nature of RNN , attention-based methods only consider the self-attention on existing edges and are sensitive to noisy graphs . In contrast , DGT leverages Transformer to capture the spatial-temporal dependency between all nodes pairs , does not over-relying on the given graph structures , and is less sensitive to noisy edges . Graph Transformers . Recently , several attempts have been made to leverage Transformer for Key graph representation learning . For example , GRAPHORMER ( Ying et al. , 2021 ) and GRAPHTRANSFORMER ( Dwivedi & Bresson , 2020 ) use scaled dot-product attention ( Vaswani et al. , 2017 ) for message aggregation and generalizes the idea of positional encoding to graph domains . GRAPHBERT ( Zhang et al. , 2020 ) first samples an egocentric network for each node , then order all nodes into a sequence based on node importance , and feed into the Transformer . However , GRAPHORMER is only feasible to small molecule graphs and can not scale to large graphs due to the significant computation cost of full attention ; GRAPHTRANSFORMER only considers the first-hop neighbor aggregation , which makes it sensitive to noisy graphs ; GRAPHBERT does not leverage the graph topology and can perform poorly when graph topology is important . In contrast , DGT encodes the input graph structures as an inductive bias to guide the full-attention optimization , which balances the trade-offs between noisy input robustness and efficiently learning an underlying graph structure . A detailed comparison is deferred to Appendix D . 3 METHOD . In this section , we first introduce the temporal union-graph ( in Section 3.1 ) and our sampling strategy ( in Section 3.2 ) that can reduce the overall complexity from the temporal and spatial perspectives respectively . Then , we introduce our spatial-temporal encoding technique ( in Section 3.3 ) , describe the two-tower transformer architecture design , and explain how to integrate the spatial-temporal encoding to DGT ( in Section 3.4 ) . Figure 1 illustrates the overall DGT design . 3.1 TEMPORAL-UNION GRAPH GENERATION . One major challenge of applying Transformers on graph representation learning is its significant computation and memory overhead . In Transformers , the computation cost of self-attention is O ( |E| d ) and its memory cost is O ( |E|+ |V| d ) . When using full attention , the computation graph is fully connected with |E| = |V|2 , where the overall complexity is quadratic in the graph size . On dynamic graphs , this problem can be even more severe if one naively extends the static graph algorithm to a dynamic graph , e.g. , first extracting the spatial information of each snapshot graph separately , then jointly reasoning the temporal information on all snapshot graphs ( Sankar et al. , 2018 ; Pareja et al. , 2020 ) . By doing so , the overall complexity grows linearly with the number of time-steps T , i.e. , with O ( |V|2Td ) computation and O ( |V|2T + |V|Td ) memory cost . To reduce the dependency of the overall complexity on the number of time-steps , we propose to first aggregate dynamic graphs G = { G1 , . . . , GT } into a temporal-union graph Gunion ( V , E ′ ) then employ DGT on the generated temporal-union graph , where E ′ = Unique { ( i , j ) : ( i , j ) ∈ Et , t ∈ [ T ] } is the set of all possible unique edges in G. As a result , the overall complexity of DGT does not grow with the number of time-steps . Details on how to leverage spatial-temporal encoding to recover the temporal information of edges are described in Section 3.3 . 3.2 TARGET NODE DRIVEN CONTEXT NODE SAMPLING . Although the temporal-union graph can alleviate the computation burden from the temporal dimension , due to the overall quadratic complexity of self-attention with respect to the input graph size , scaling the training of Transformer to real-world graphs is still non-trivial . Therefore , a properly designed sampling strategy that makes the overall complexity independent with graph sizes is necessary . Our goal is to design a sub-graph sampling strategy that ensures a fixed number of well-connected nodes and a lower computational complexity . To this end , we propose to first sample a subset of nodes that we are interested in as target nodes , then sample their common neighbors as context nodes . Let target nodes Vtgt ⊆ V be the set of nodes that we are interested in and want to compute its node representation . For example , for the link prediction task , Vtgt are the set of nodes that we aim to predict whether they are connected . Then , the context nodes Vctx ⊆ { N ( i ) | ∀i ∈ Vtgt } are sampled as the common neighbors of the target nodes . Notice that since context nodes Vctx are sampled as the common neighbors of the target nodes , they can provide local structure information for nodes in the target node set . Besides , since two different nodes in the target node set can be far apart with a disconnected neighborhood , the neighborhood of two nodes can provide an approximation of the global view of the full graph . During the sampling process , to control the randomness involved in the sampling process , Vctx are chosen as the subset of nodes with the top-K joint Personalized PageRank ( PPR ) score ( Andersen et al. , 2006 ) to nodes in Vtgt , where PPR score is a node proximity measure that captures the importance of two nodes in the graph . More specifically , our joint PPR sampler proceeds as follows : First , we compute the approximated PPR vector π ( i ) ∈ RN for all node i ∈ Vtgt , where the j-th element in π ( i ) can be interpreted as the probability of a random walk to start at node i and end at node j . We then compute the approximated joint PPR vector π̂ ( Vtgt ) = ∑ i∈Vtgt π ( i ) ∈ R N . Finally , we select K context nodes where each node j ∈ Vctx has the top-K joint PPR score in π̂ ( Vtgt ) . In practice , we select the context node size the same as the target node size , i.e. , K = |Vtgt| .

Summary: This paper makes an attempt to study the dynamic graph representation learning. The paper proposes a couple tricks for the combination with graph transformer networks. The tricks include sampling, union graph, pretraining etc. Some experiments have been conducted.