DyRep: Learning Representations over Dynamic Graphs

1 INTRODUCTION . Representation learning over graph structured data has emerged as a keystone machine learning task due to its ubiquitous applicability in variety of domains such as social networks , bioinformatics , natural language processing , and relational knowledge bases . Learning node representations to effectively encode high-dimensional and non-Euclidean graph information is a challenging problem but recent advances in deep learning has helped important progress towards addressing it ( Cao et al. , 2015 ; Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ; Tang et al. , 2015 ; Wang et al. , 2016a ; 2017 ; Xu et al. , 2017 ) , with majority of the approaches focusing on advancing the state-of-the-art in static graph setting . However , several domains now present highly dynamic data that exhibit complex temporal properties in addition to earlier cited challenges . For instance , social network communications , financial transaction graphs or longitudinal citation data contain fine-grained temporal information on nodes and edges that characterize the dynamic evolution of a graph and its properties over time . These recent developments have created a conspicuous need for principled approaches to advance graph embedding techniques for dynamic graphs ( Hamilton et al. , 2017b ) . We focus on two pertinent questions fundamental to representation learning over dynamic graphs : ( i ) What can serve as an elegant model for dynamic processes over graphs ? — A key modeling choice in existing representation learning techniques for dynamic graphs ( Goyal et al. , 2017 ; Zhou et al. , 2018 ; Trivedi et al. , 2017 ; Ngyuyen et al. , 2018 ; Yu et al. , 2018 ) assume that graph dynamics evolve as a single time scale process . In contrast to these approaches , we observe that most real-world graphs exhibit at least two distinct dynamic processes that evolve at different time scales — Topological Evolution : where the number of nodes and edges are expected to grow ( or shrink ) over time leading to structural changes in the graph ; and Node Interactions : which relates to activities between nodes that may or may not be structurally connected . Modeling interleaved dependencies between these non-linearly evolving dynamic processes is a crucial next step for advancing the formal models of dynamic graphs . ∗Corresponding Author : { rstrivedi @ gatech.edu , farajtabar @ google.com , zha @ cc.gatech.edu } ( ii ) How can one leverage such a model to learn dynamic node representations that are effectively able to capture evolving graph information over time ? — Existing techniques in this direction can be divided into two approaches : a . ) Discrete-Time Approach , where the evolution of a dynamic graph is observed as collection of static graph snapshots over time ( Zhu et al. , 2016 ; Goyal et al. , 2017 ; Zhou et al. , 2018 ) . These approaches tend to preserve ( encode ) very limited structural information and capture temporal information at a very coarse level which leads to loss of information between snapshots and lack of ability to capture fine-grained temporal dynamics . Another challenge in such approaches is the selection of appropriate aggregation granularity which is often misspecified . b . ) Continuous-Time Approach , where evolution is modeled at finer time granularity in order to address the above challenges . While existing approaches have demonstrated to be very effective in specific settings , they either model simple structural and complex temporal properties in a decoupled fashion ( Trivedi et al. , 2017 ) or use simple temporal models ( exponential family in ( Ngyuyen et al. , 2018 ) ) . But several domains exhibit highly nonlinear evolution of structural properties coupled with complex temporal dynamics and it remains an open problem to effectively model and learn informative representations capturing various dynamical properties of such complex systems . As noted in ( Chazelle , 2012 ) , an important requirement to effectively learn over such dynamical systems is the ability to express the dynamical processes at different scales . We propose that any dynamic graph must be minimally expressed as a result of two fundamental processes evolving at different time scales : Association Process ( dynamics of the network ) , that brings change in the graph structure and leads to long lasting information exchange between nodes ; and Communication Process ( dynamics on the network ) , that relates to activities between ( not necessarily connected ) nodes which leads to temporary information flow between them ( Farine , 2017 ; Artime et al. , 2017 ) . We , then , posit our goal of learning node representations as modeling a latent mediation process that bridges the above two observed processes such that learned representations drive the complex temporal dynamics of both processes and these processes subsequently lead to the nonlinear evolution of node representations . Further , the information propagated across the graph is governed by the temporal dynamics of communication and association histories of nodes with its neighborhood . For instance , in a social network , when a node ’ s neighborhood grows , it changes that node ’ s representation which in turn affects her social interactions ( association→ embedding→ communication ) . Similarly , when node ’ s interaction behavior changes , it affects the representation of her neighbors and herself which in turn changes the structure and strength of her connections due to link addition or deletion ( communication→ embedding→ association ) . We call this phenomenon — evolution through mediation and illustrate it graphically in Figure 1 . In this work , we propose a novel representation learning framework for dynamic graphs , DyRep , to model interleaved evolution of two observed processes through latent mediation process expressed above and effectively learn richer node representations over time . Our framework ingests dynamic graph information in the form of association and communication events over time and updates the node representations as they appear in these events . We build a two-time scale deep temporal point process approach to capture the continuous-time fine-grained temporal dynamics of the two observed processes . We further parameterize the conditional intensity function of the temporal point process with a deep inductive representation network that learns functions to compute node representations . Finally , we couple the structural and temporal components of our framework by designing a novel Temporal Attention Mechanism , which induces temporal attentiveness over neighborhood nodes using the learned intensity function . This allows to capture highly interleaved and nonlinear dynamics governing node representations over time . We design an efficient unsupervised training procedure for end-to-end training of our framework . We demonstrate consistent and significant improvement over state-of-the-art representative baselines on two real-world dynamic graphs for the tasks of dynamic link prediction and time prediction . We further present an extensive qualitative analysis through embedding visualization and ablation studies to discern the effectiveness of our framework . 2 BACKGROUND AND PRELIMINARIES . 2.1 RELATED WORK . Representation Learning approaches for static graphs either perform node embedding ( Cao et al. , 2015 ; Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ; Tang et al. , 2015 ; Wang et al. , 2016a ; 2017 ; Xu et al. , 2017 ) or sub-graph embedding ( Scarselli et al. , 2009 ; Li et al. , 2016 ; Dai et al. , 2016 ) which can also utilize convolutional neural networks ( Kipf & Welling , 2017 ; 2016 ; Bruna et al. , 2014 ) . Among them , GraphSage ( Hamilton et al. , 2017a ) is an inductive method for learning functions to compute node representations that can be generalized to unseen nodes . Most of these approaches only work with static graphs or can model evolving graphs without temporal information . Dynamic network embedding is pursued through various techniques such as matrix factorization ( Zhu et al. , 2016 ) , structural properties ( Zhou et al. , 2018 ) , CNN-based approaches ( Seo et al. , 2016 ) , deep recurrent models ( Trivedi et al. , 2017 ) , and random walks ( Ngyuyen et al. , 2018 ) . There exists a rich body of literature on temporal modeling of dynamic networks ( Kim et al. , 2017 ) , that focus on link prediction tasks but their goal is orthogonal to our work as they build task specific methods and do not focus on representation learning . Authors in ( Yang et al. , 2017 ; Sarkar et al. , 2007 ) proposed models of learning dynamic embeddings but none of them consider time at finer level and do not capture both topological evolution and interactions simultaneously . In parallel , research on deep point process models include parametric approaches to learn intensity ( Du et al. , 2016 ; Mei & Eisner , 2017 ) using recurrent neural networks and GAN based approaches to learn intensity functions ( Xiao et al. , 2017 ) . More detailed related works are provided in Appendix F . 2.2 TEMPORAL POINT PROCESSES . Stochastic point processes ( Daley & Vere-Jones , 2007 ) are random processes whose realization comprises of discrete events in time , t1 , t2 , . . .. A temporal point process is one such stochastic process that can be equivalently represented as a counting process , N ( t ) , which contains the number of events up to time t. The common way to characterize temporal point processes is via the conditional intensity function λ ( t ) , a stochastic model of rate of happening events given the previous events . Formally , λ ( t ) dt is the conditional probability of observing an event in the tiny window [ t , t+ dt ) , λ ( t ) dt : = P [ event in [ t , t+ dt ) |T ( t ) ] = E [ dN ( t ) |T ( t ) ] , where T ( t ) = tk|tk < t is history until t. Similarly , for t > tn and given history T = t1 , . . . , tn , we characterize the conditional probability that no event happens during [ tn , t ) as S ( t|T ) = exp ( − ∫ t tn λ ( τ ) dτ ) , which is called survival function of the process ( Aalen et al. , 2008 ) . Moreover , the conditional density that an event occurs at time t is defined as f ( t ) = λ ( t ) S ( t ) . The intensity λ ( t ) is often designed to capture phenomena of interests – common forms include Poisson Process , Hawkes processes ( Farajtabar et al. , 2014 ; Hawkes , 1971 ; Wang et al. , 2016b ; Tabibian et al. , 2017 ) , Self-Correcting Process ( Isham & Westcott , 1979 ) . Temporal Point Processes have previously been used to model both – dynamics on the network ( Farajtabar et al. , 2016 ; Zarezade et al. , 2017 ; Farajtabar et al. , 2017 ) and dynamics of the network ( Tran et al. , 2015 ; Farajtabar et al. , 2015 ) . 2.3 NOTATIONS AND DYNAMIC GRAPH SETTING . Notations . Let Gt = ( Vt , Et ) denote graph G at time t , where Vt is the set of nodes and Et is the set of edges in Gt and the edges are undirected . Event Observation – Both communication and association processes are realized in the form of dyadic events observed between nodes on graph G over a temporal window [ t0 , T ] and ordered by time . We use the following canonical tuple representation for any type of event at time t of the form e = ( u , v , t , k ) , where u , v are the two nodes involved in an event . t represents time of the event . k ∈ { 0 , 1 } and we use k = 0 to signify events from the topological evolution process ( association ) and k = 1 to signify events from node interaction process ( communication ) . Persistent edges in the graph only appear through topological events while interaction events do not contribute them . Hence , k represents an abstraction of scale ( evolution rate ) associated with processes that generate topological ( dynamic of the network ) and interaction events ( dynamic on the network ) respectively . We then represent complete set of P observed events ordered by time in window [ 0 , T ] asO = { ( u , v , t , k ) p } Pp=1 . Here , tp ∈ R+ , 0 ≤ tp ≤ T . Appendix B discusses a marked point process view of such an event set . Node Representation— Let zv ∈ Rd represent d-dimensional representation of node v. As the representation evolve over time , we qualify them as function of time : zv ( t ) — the representation of node v being updated after an event involving v at time t. We use zv ( t̄ ) for most recently updated embedding of node v just before t. Dynamic Graph Setting . Let Gt0 = ( Vt0 , Et0 ) be the initial snapshot of a graph at time t0 . Please note that Gt0 may be empty or it may contain an initial structure ( association edges ) but it will not have any communication history . Our framework observes evolution of graph as a stream of events O and hence any new node will always be observed as a part of such an event . This will induce a natural ordering over nodes as available from the data . As our method is inductive , we never learn node-specific representations and rather learn functions to compute node representations . In this work , we only support growth of network i.e . we only model addition of nodes and structural edges and leave deletion as future work . Further , for general description of the model , we will assume that an edge in the graph do not have types and nodes do not have attributes but we discuss the details on how to use our model to accommodate these features in Appendix B .

Summary: Overall, the contribution of the paper is somewhat limited [but a little more than my initial assessment, thanks to the rebuttal]. It is essentially an extension of (Trivedi et al. 2017), adding attention to provide self-exciting rates, applied to two types of edges (communication edges and “friendship” edges). Conditioned on past edges, future edges are assumed independent, which makes the math trivial. The work would be better described as modeling a Marked Point Process with marks k \in {0,1}.