Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks

1 INTRODUCTION . Temporal networks consider dynamically interacting elements as nodes , interactions as temporal links , with labels of when those interactions happen . Such temporal networks provide abstractions to study many real-world dynamic systems ( Holme & Saramäki , 2012 ) . Researchers have investigated temporal networks in recent several decades and concluded many insightful laws that essentially reflect how these real-world systems evolve over time ( Kovanen et al. , 2011 ; Benson et al. , 2016 ; Paranjape et al. , 2017 ; Zitnik et al. , 2019 ) . For example , the law of triadic closure in social networks , describing that two nodes with common neighbors tend to have a mutual interaction later , reflects how people establish social connections ( Simmel , 1950 ) . Later , a more elaborate law on the correlation between the interaction frequency between two individuals and the degree that they share social connections , further got demonstrated ( Granovetter , 1973 ; Toivonen et al. , 2007 ) . Feedforward control loops that consist of a direct interaction ( from node w to node u ) and an indirect interaction ( from w through another node v to u ) , also work as a law in the modulation of gene regulatory systems ( Mangan & Alon , 2003 ) and also as the control principles of many engineering systems ( Gorochowski et al. , 2018 ) . Although research on temporal networks has achieved the above success , it can hardly be generalized to study more complicated laws : Researchers have to investigate an exponentially increasing number of patterns when incorporating more interacting elements let alone their time-evolving aspects . Recently , representation learning , via learning vector representations of data based on neural networks , has offered unprecedented possibilities to extract , albeit implicitly , more complex structural patterns ( Hamilton et al. , 2017b ; Battaglia et al. , 2018 ) . However , as opposed to the study on static networks , representation learning of temporal networks is far from mature . Two challenges on temporal networks have been frequently discussed . First , the entanglement of structural and temporal ∗Project website with code and data : http : //snap.stanford.edu/caw/ 𝑡3 𝑤 Triadic closure ( 𝑡1 , 𝑡2 < 𝑡3 ) Law explanation : Two nodes that share a common node directly connected to them tend to be connected . uw vw 𝑢 𝑣 𝑤 𝑢 𝑣 𝑤 Feed-forward control ( 𝑡1 < 𝑡2 < 𝑡3 ) 𝑢 𝑣 𝑤 𝑢 𝑣 𝑡1 𝑡2 𝑡1 𝑡2 𝑡1 𝑡2 𝑡1 𝑡2 𝑡3 𝑡1 𝑡2 Law explanation : One node ( w ) activates another node first ( u ) and then activates a third node ( v ) . Mostly the third node will inhibit the second node later . uw vw 𝑡1 𝑡2 CAWs extracted : CAWs extracted : , , Figure 1 : Triadic closure and feed-forward loops : Causal anonymous walks ( CAW ) capture the laws . Example : three 3-step walks ( 𝑡𝑥 , 𝑋 are the default timestamp and the default node when no historical links can be found ) Backtrack m-step random walks over time before t=10 : u b a c 6 3 1 𝑀 walks starting at 𝒖 𝒗𝒖 𝑡 = 10 ? 0,3,8 3,6 4 , 5 3,9 7,9 2 4,88 9 e d b c a g h1 0,3,7 … v e u b 8 7 6 𝑀 walks starting at 𝒗 … u b a c 6 3 1 u c b a 5 3 0 3 u b a x3 0 𝑡𝑥 v e u b 8 7 6 v d u3 2 v g h v 9 7 4 x𝑆𝑣 : 𝑆𝑢 : 0 , 2 , 1 , 0 𝑇 𝑡𝑥 u b a c 16 3 6 𝐼𝐶𝐴𝑊 𝑏𝐼𝐶𝐴𝑊 𝑢 3 𝐼𝐶𝐴𝑊 𝑎 1 𝐼𝐶𝐴𝑊 𝑐 Anonymize : Causality Extraction Set-based Anonymization ✓ ✓ ✓ ✓ 0 , 0 , 0 , 1 𝑇 Count number of 𝑏 ’ s in different positions : 𝐼𝐶𝐴𝑊 𝑏 ; 𝑆𝑢 , 𝑆𝑣 = 𝑔 𝑏 ; 𝑆𝑢 , 𝑔 𝑏 ; 𝑆𝑣 ( Relative node identity ) A temporal graph with timestamped links and a queried link at certain time : Figure 2 : Causal anonymous walks ( CAW ) : causality extraction and set-based anonymization . patterns required an elegant model to digest the two-side information . Second , the model scalability becomes more crucial over temporal networks as new arriving links need to be processed timely while a huge link set due to the repetitive links between two nodes needs to be digested simultaneously . In contrast to the above two challenges , another challenge , the inductive capability of the temporalnetwork representation , is often ignored . However , it is equally important if not more , as the inductive capability indicates whether the models indeed capture the dynamic laws of the systems and can be further generalized to the system that share the same laws but have not been used to train these models . These laws may only depend on structures such as the triadic closure or feed-forward control loops as aforementioned . These laws may also correlate with node attributes , such as interactions between people affected by their gender and age ( Kovanen et al. , 2013 ) . But in both cases , the laws should be independent from network node identities . Although previous works tend to learn inductive models by removing node identities ( Trivedi et al. , 2019 ; Xu et al. , 2020 ) , they run into other issues to inductively represent the dynamic laws , for which we leave more detailed discussion in Sec . 2 . Here we propose Causal Anonymous Walks ( CAW ) for modeling temporal networks . Our idea for inductive learning is inspired by the recent investigation on temporal network motifs that correspond to connected subgraphs with links that appear within a restricted time range ( Kovanen et al. , 2011 ; Paranjape et al. , 2017 ) . Temporal network motifs essentially reflect network dynamics : Both triadic closure and feed-forward control can be viewed as temporal network motifs evolving ( Fig . 1 ) ; An inductive model should predict the 3rd link in both cases when it captures the correlation of these two links as they share a common node , while the model is agnostic to the node identities of these motifs . Our CAW model has two important properties ( Fig . 2 ) : ( 1 ) Causality extraction — a CAW starts from a link of interest and backtracks several adjacent links over time to encode the underlying causality of network dynamics . Each walk essentially gives a temporal network motif ; ( 2 ) Set-based anonymization — CAWs remove the node identities over the walks to guarantee inductive learning while encoding relative node identities based on the counts that they appear at a certain position according to a set of sampled walks . Relative node identities guarantee that the structures of motifs and their correlations are still kept after removing node identities . To predict temporal links between two nodes of interest , we propose a model CAW-Network ( CAW-N ) that samples a few CAWs related to the two nodes of interest , encodes and aggregates these CAWs via RNNs ( Rumelhart et al. , 1986 ) and set-pooling respectively to make the prediction . Experiments show that CAW-N is extremely effective . CAW-N does not need to enumerate the types of motifs and count their numbers that have been used as features to predict network dynamics ( Lahiri & Berger-Wolf , 2007 ; Rahman & Al Hasan , 2016 ; Rossi et al. , 2019 ; AbuOda et al. , 2019 ; Li & Milenkovic , 2017 ) , which significantly saves feature-engineering effort . CAW-N also keeps all fine-grained temporal information along the walks that may be removed by directly counting motifs ( Ahmed et al. , 2015 ; Paranjape et al. , 2017 ) . CAWs share a similar idea as anonymous walks ( AW ) ( Micali & Zhu , 2016 ) to remove node identities . However , AWs have only been used for entire static graph embedding ( Ivanov & Burnaev , 2018 ) and are not directedly applied to represent temporal networks : AWs can not capture causality ; AWs get anonymized based on each single walk and hence lose the correlation between network motifs . In contrast , CAWs capture all the information , temporal , structural , motif-correlation that are needed , to represent temporal networks . We conclude our contributions in three-folds : ( 1 ) A novel approach to represent temporal network CAW-N is proposed , which leverages CAWs to encode temporal network motifs to capture network dynamics while keeping fully inductive . CAW-N is evaluated to predict links over 6 real-world temporal networks . CAW-N outperforms all SOTA methods by about 15 % averaged over 6 networks in the inductive setting and also significantly beat all SOTA methods over 5 networks in the transductive setting ; ( 2 ) CAW-N significantly decreases the feature-engineering effort in traditional motif selection and counting approaches and keeps fine-grained temporal information ; ( 3 ) CAW-N is paired with a CAW sampling method with constant memory and time cost , which conduces to online learning . 2 RELATED WORK . Prior work on representation learning of temporal networks preprocesses the networks by simply aggregating the sequence of links within consecutive time windows into network snapshots , and use graph neural networks ( GNN ) ( Scarselli et al. , 2008 ; Kipf & Welling , 2017 ) and RNNs or transformer networks ( Vaswani et al. , 2017 ) to encode structural patterns and temporal patterns respectively ( Pareja et al. , 2020 ; Manessi et al. , 2020 ; Goyal et al. , 2020 ; Hajiramezanali et al. , 2019 ; Sankar et al. , 2020 ) . The main drawback of these approaches is that they need to predetermine a time granularity for link aggregation , which is hard to learn structural dynamics in different time scales . Therefore , approaches that work on link streams directly have been recently proposed ( Trivedi et al. , 2017 ; 2019 ; Kumar et al. , 2019 ; Xu et al. , 2020 ) . Know-E ( Trivedi et al. , 2017 ) , DyRep ( Trivedi et al. , 2019 ) and JODIE ( Kumar et al. , 2019 ) use RNNs to propagate messages across interactions to update node representations . Know-E , JODIE consider message exchanges between two directly interacted nodes while DyRep considers an additional hop of interactions . Therefore , DyRep gives a more expressive model at a cost of high complexity . TGAT ( Xu et al. , 2020 ) in contrast mimics GraphSAGE ( Hamilton et al. , 2017a ) and GAT ( Veličković et al. , 2018 ) to propagate messages in a GNN-like way from sampled historical neighbors of a node of interest . TGAT ’ s sampling strategy requires to store all historical neighbors , which is unscalable for online learning . Our CAW-N directly works on link streams and only requires to memorize constant many most recent links for each node . Most of the above models are not inductive because they associate each node with an onehot identity ( or the corresponding row of the adjacency matrix , or a free-trained vector ) ( Li et al. , 2018 ; Chen et al. , 2019 ; Kumar et al. , 2019 ; Hajiramezanali et al. , 2019 ; Sankar et al. , 2020 ; Manessi et al. , 2020 ; Goyal et al. , 2020 ) . TGAT ( Xu et al. , 2020 ) claimed to be inductive by removing node identities and just encoding link timestamps and attributes . However , TGAT was only evaluated over networks with rich link attributes , where the structural dynamics is not captured essentially : If we focus on structural dynamics only , it is easy to show a case when TGAT confuses node representations and will fail : Suppose in the history , two node pairs { a , b } and { a′ , b′ } only interact within each pair but share the timestamps ( Fig . 3 ) . Intuitively , a proper model should predict that future links still appear within each pair . However , TGAT can not distinguish a v.s . a′ , and b v.s . b′ , which leads to incorrect prediction . Note that GraphSAGE ( Hamilton et al. , 2017a ) and GAT ( Veličković et al. , 2018 ) also share the similar issue when representing static networks for link prediction ( Zhang et al. , 2020 ; Srinivasan & Ribeiro , 2019 ) . DyRep ( Trivedi et al. , 2019 ) is able to relieve such ambiguity by merging node representations with their neighbors ’ via RNNs . However , when DyRep runs over a new network , it frequently encounters node representations unseen during its training and will fail to make correct prediction . 6 Return { Wi|1 ≤ i ≤M } ; Our CAW-N removes node identities and leverages relative node identities to avoid the issue in Fig . 3 . Detailed explanations are given in Sec.4.2 . Network-embedding approaches may also be applied to temporal networks ( Zhou et al. , 2018 ; Du et al. , 2018 ; Mahdavi et al. , 2018 ; Singer et al. , 2019 ; Nguyen et al. , 2018 ) . However , they directly assign each node with a learnable vector . Therefore , they are not inductive and can not digest attributes .

Summary: The authors provide in-depth analysis on the critical topic of capturing dynamic laws for the inductive representation learning of temporal graphs. The authors leverage the causal anonymous walk to capture the topological laws of the dynamic graph, while not requiring to memorize node identities such that inductive learning is still feasible. Compared with the previous work, the paper emphasizes on scenarios where node/edge attributes are less informative of the inductive reasoning. Some workaround methods are also proposed to deal with the edge features and to enable the efficient computation and sampling procedure.