Dynamic Graph Representation Learning via Self-Attention Networks

1 INTRODUCTION . Learning latent representations ( or embeddings ) of nodes in graphs has been recognized as a fundamental learning problem due to its widespread use in various domains such as social media ( Perozzi et al. , 2014 ) , biology ( Grover & Leskovec , 2016 ) , and knowledge bases ( Wang et al. , 2014 ) . The basic idea is to learn a low-dimensional vector for each node , which encodes the structural properties of a node and its neighborhood ( and possibly attributes ) . Such low-dimensional representations can benefit a plethora of graph analytical tasks such as node classification , link prediction , and graph visualization ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ; Wang et al. , 2016 ) . Previous work on graph representation learning mainly focuses on static graphs , which contain a fixed set of nodes and edges . However , many graphs in real-world applications are intrinsically dynamic , in which graph structures can evolve over time . They are usually represented as a sequence of graph snapshots from different time steps ( Leskovec et al. , 2007 ) . Examples include academic co-authorship networks where authors may periodically switch their collaboration behaviors and email communication networks whose structures may change dramatically due to sudden events . In such scenarios , modeling temporal evolutionary patterns is important in accurately predicting node properties and future links . Learning dynamic node representations is challenging , compared to static settings , due to the complex time-varying graph structures : nodes can emerge and leave , links can appear and disappear , and communities can merge and split . This requires the learned embeddings not only to preserve structural proximity of nodes , but also to jointly capture the temporal dependencies over time . Though some recent work attempts to learn node representations in dynamic graphs , they mainly impose a temporal regularizer to enforce smoothness of the node representations from adjacent snapshots ( Zhu et al. , 2016 ; Li et al. , 2017 ; Zhou et al. , 2018 ) . However , these approaches may fail when nodes exhibit significantly distinct evolutionary behaviors . Trivedi et al . ( 2017 ) employ a recurrent neural architecture for temporal reasoning in multi-relational knowledge graphs . However , their temporal node representations are limited to modeling first-order proximity , while ignoring the structure of higher-order graph neighborhoods . Attention mechanisms have recently achieved great success in many sequential learning tasks such as machine translation ( Bahdanau et al. , 2015 ) and reading comprehension ( Yu et al. , 2018 ) . The key underlying principle is to learn a function that aggregates a variable-sized input , while focusing on the parts most relevant to a certain context . When the attention mechanism uses a single sequence as both the inputs and the context , it is often called self-attention . Though attention mechanisms were initially designed to facilitate Recurrent Neural Networks ( RNNs ) to capture long-term dependencies , recent work by Vaswani et al . ( 2017 ) demonstrates that a fully self-attentional network itself can achieve state-of-the-art performance in machine translation tasks . Velickovic et al . ( 2018 ) extend self-attention to graphs by enabling each node to attend over its neighbors , achieving state-of-the-art results for semi-supervised node classification tasks in static graphs . As dynamic graphs usually include periodical patterns such as recurrent links or communities , attention mechanisms are capable of utilizing information about most relevant historical context , to facilitate future prediction . Inspired by recent work on attention techniques , we present a novel neural architecture named Dynamic Self-Attention Network ( DySAT ) to learn node representations on dynamic graphs . Specifically , we employ self-attention along two dimensions : structural neighborhoods and temporal dynamics , i.e. , DySAT generates a dynamic representation for a node by considering both its neighbors and historical representations , following a self-attentional strategy . Unlike static graph embedding methods that focus entirely on preserving structural proximity , we learn dynamic node representations that reflect the temporal evolution of graph structure over a varying number of historical snapshots . In contrast to temporal smoothness-based methods , DySAT learns attention weights that capture temporal dependencies at a fine-grained node-level granularity . We evaluate our framework on the dynamic link prediction task using four benchmarks of different sizes including two email communication networks ( Klimt & Yang , 2004 ; Panzarasa et al. , 2009 ) and two bipartite rating networks ( Harper & Konstan , 2016 ) . Our evaluation results show that DySAT achieves significant improvements ( 3.6 % macro-AUC on average ) over several stateof-the-art baselines and maintains a more stable performance over different time steps . 2 RELATED WORK . Our framework is related to previous representation learning techniques on static graphs , dynamic graphs , and recent developments in self-attention mechanisms . Static graph embeddings . Early work on unsupervised graph representation learning exploits the spectral properties of various graph matrix representations , such as Laplacian , etc . to perform dimensionality reduction ( Tenenbaum et al. , 2000 ; Belkin & Niyogi , 2001 ) . To improve scalability , some work ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) utilizes Skip-gram methods , inspired by their success in Natural Language Processing ( NLP ) . Recently , several graph neural network architectures based on generalizations of convolutions have achieved tremendous success , among which many methods are designed for supervised or semi-supervised learning tasks ( Niepert et al. , 2016 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Sankar et al. , 2017 ; Velickovic et al. , 2018 ) . Hamilton et al . ( 2017b ) extend graph convolutional methods through trainable neighborhood aggregation functions , to propose a general framework applicable to unsupervised representation learning . However , these methods are not designed to model temporal evolutionary patterns in dynamic graphs . Dynamic graph embeddings . Most techniques employ temporal smoothness regularization to ensure embedding stability across consecutive time-steps ( Zhu et al. , 2016 ) . Zhou et al . ( 2018 ) additionally use triadic closure ( Kossinets & Watts , 2006 ) as guidance , leading to significant improvements . Neural methods were recently explored in the knowledge graph domain by Trivedi et al . ( 2017 ) , who employ a recurrent neural architecture for temporal reasoning . However , their model is limited to tracing link evolution , thus limited to capturing first-order proximity . Goyal et al . ( 2017 ) learn incremental node embeddings through initialization from the previous time steps , however , this may not guarantee the model to capture long-term graph similarity . A few recent works ( Nguyen et al. , 2018 ; Zuo et al. , 2018 ) examine a related setting of temporal graphs with continuous timestamped links for representation learning , which is however orthogonal to the established problem setup of using dynamic graph snapshots . Li et al . ( 2017 ) learn node embeddings in dynamic attributed graphs by initially training an offline model , followed by incremental updates over time . However , their key focus is online learning to improve efficiency over re-training static models , while our goal is to improve representation quality by exploiting the temporal evolutionary patterns in graph structure . Unlike previous approaches , our framework captures the most relevant historical contexts through a self-attentional architecture , to learn dynamic node representations . Self-attention mechanisms . Recent advancements in many NLP tasks have demonstrated the superiority of self-attention in achieving state-of-the-art performance ( Vaswani et al. , 2017 ; Lin et al. , 2017 ; Tan et al. , 2018 ; Shen et al. , 2018 ; Shaw et al. , 2018 ) . In DySAT , we employ self-attention mechanisms to compute a dynamic node representation by attending over its neighbors and previous historical representations . Our approach of using self-attention over neighbors is closely related to the Graph Attention Network ( GAT ) ( Velickovic et al. , 2018 ) , which employs neighborhood attention for semi-supervised node classification in a static graph . As dynamic graphs usually contain periodical patterns , we extend the self-attention mechanisms over the historical representations of a particular node to capture its temporal evolution behaviors . 3 PROBLEM DEFINITION . In this work , we address the problem of dynamic graph representation learning . A dynamic graph is defined as a series of observed snapshots , G = { G1 , . . . , GT } where T is the number of time steps . Each snapshot Gt = ( V , Et ) is a weighted undirected graph with a shared node set V , a link set Et , and weighted adjacency matrix At at time t. Unlike some previous work that assumes links can only be added over time in dynamic works , we also allow to remove links . Dynamic graph representation learning aims to learn latent representations etv ∈ Rd for each node v ∈ V at time steps t = 1 , 2 , . . . , T , such that etv preserves both the local graph structures centered at v and its evolutionary behaviors prior to time t . 4 DYNAMIC SELF-ATTENTION NETWORK . In this section , we first describe the high-level structure of our model . DySAT consists of two major novel components : structural and temporal self-attention layers , which can be utilized to construct arbitrary graph neural architectures through stacking of layers . Similar to existing studies on attention mechanisms , we employ multi-head attentions to improve model capacity and stability . DySAT consists of a structural block followed by a temporal block , as illustrated in Figure 1 , where each block may contain multiple stacked layers of the corresponding layer type . The structural block extracts features from the local neighborhood through self-attentional aggregation , to compute intermediate node representations for each snapshot . These representations feed as input to the temporal block , which attends over multiple time steps , capturing temporal variations in the graph . 4.1 STRUCTURAL SELF-ATTENTION . The input of this layer is a graph snapshot G ∈ G and a set of input node representations { xv ∈ RD , ∀v ∈ V } where D is the input embedding dimension . The input to the initial layer can be set as 1-hot encoded vectors for each node ( or attributes if available ) . The output is a new set of node representations { zv ∈ RF , ∀v ∈ V } with F dimensions that capture local structural properties . Specifically , the structural self-attention layer attends over the immediate neighbors of a node v ( in snapshot G ) , by computing attention weights as a function of their input node embeddings . The structural attention layer is a variant of GAT ( Velickovic et al. , 2018 ) , applied on a single snapshot : zv = σ ( ∑ u∈Nv αuvW sxu ) , αuv = exp ( σ ( Auv · aT [ W sxu||W sxv ] ) ) ∑ w∈Nv exp ( σ ( Awv · aT [ W sxw||W sxv ] ) ) ( 1 ) where Nv = { u ∈ V : ( u , v ) ∈ E } is the set of immediate neighbors of node v in snapshot G ; W s ∈ RD×F is a shared weight transformation applied to each node in the graph ; a ∈ R2D is a weight vector parameterizing the attention function implemented as feed-forward layer ; || is the concatenation operation and σ ( · ) is a non-linear activation function . Note that Auv is the weight of link ( u , v ) in the current snapshot G. The set of learned coefficients αuv , obtained by a softmax over the neighbors of each node , indicate the importance or contribution of node u to node v at the current snapshot . We use a LeakyRELU non-linearity to compute the attention weights , followed by ELU for the output representations . In our experiments , we employ sparse matrices to implement the masked self-attention over neighbors .

Summary: This paper proposes a model for learning node embedding vectors of dynamic graphs, whose edge topology may change. The proposed model, called Dynamic Self-Attention Network (DySAT), uses attention mechanism to represent the interaction of spatial neighbouring nodes, which is closely related to the Graph Attention Network. For the temporal dependency between successive graphs, DySAT also uses attention structure inspired by previous work in machine translation. Experiments on 4 datasets show that DySAT can improve the AUC of link prediction by significant margins, compared to static graph methods and other dynamic graph methods. Though the attention structures in this paper are not original, combining these structures and applying them on dynamic graph embedding is new.