Time-varying Graph Representation Learning via Higher-Order Skip-Gram with Negative Sampling

1 INTRODUCTION . A great variety of natural and artificial systems can be represented as networks of elementary structural entities coupled by relations between them . The abstraction of such systems as networks helps us understand , predict and optimize their behaviour ( Newman , 2003 ; Albert & Barabási , 2002 ) . In this sense , node and graph embeddings have been established as standard feature representations in many learning tasks ( Cai et al. , 2018 ; Goyal & Ferrara , 2018 ) . Node embedding methods map nodes into low-dimensional vectors that can be used to solve downstream tasks such as edge prediction , network reconstruction and node classification . Node embeddings have proven successful in achieving low-dimensional encoding of static network structures , but many real-world networks are inherently dynamic ( Holme & Saramäki , 2012 ) . Timeresolved networks are also the support of important dynamical processes , such as epidemic or rumor spreading , cascading failures , consensus formation , etc . ( Barrat et al. , 2008 ) . Time-resolved node embeddings have been shown to yield improved performance for predicting the outcome of dynamical processes over networks , such as information diffusion and disease spreading ( Sato et al. , 2019 ) , providing estimation of infection and contagion risk when used with contact tracing data . Since we expect having more data on proximity networks being used for contact tracing and as proxies for epidemic risk ( Alsdurf et al. , 2020 ) , learning meaningful representations of time-resolved proximity networks can be of extreme importance when facing events such as epidemic outbreaks ( Kapoor et al. , 2020 ; Gao et al. , 2020 ) . The manual and automatic collection of time-resolved proximity graphs for contact tracing purposes presents an opportunity for quick identification of possible infection clusters and infection chains . Even before the COVID-19 pandemic , the use of wearable proximity sensors for collecting time-resolved proximity networks has been largely discussed in the literature and many approaches have been used to describe patterns of activity and community structure , and to study spreading patterns of infectious diseases ( Sapienza et al. , 2015 ; Gauvin et al. , 2014 ; Génois et al. , 2015 ) . Here we propose a representation learning model that performs implicit tensor factorization on different higher-order representations of time-varying graphs . The main contributions are as follows : Given that the skip-gram embedding approach implicitly performs a factorization of the shifted pointwise mutual information matrix ( PMI ) ( Levy & Goldberg , 2014 ) , we generalize it to perform implicit factorization of a shifted PMI tensor . We then define the steps to achieve this factorization using higher-order skip-gram with negative sampling ( HOSGNS ) optimization . We show how to apply 3rd-order and 4th-order SGNS on different higher-order representations of time-varying graphs . Finally , we show that time-varying graph representations learned via HOSGNS outperform stateof-the-art methods when used to solve downstream tasks , even using a fraction of the number of embedding parameters . We report the results of learning embeddings on empirical time-resolved face-to-face proximity data and using such representations as predictors for solving two different tasks : network reconstruction and predicting the outcomes of a SIR spreading process over the time-varying graph . We compare these results with state-of-the art methods for time-varying graph representation learning . 2 PRELIMINARIES AND RELATED WORK . Skip-gram representation learning . The skip-gram model was designed to compute word embeddings in WORD2VEC ( Mikolov et al. , 2013 ) , and afterwards extended to graph node embeddings ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ) . Levy & Goldberg ( 2014 ) established the relation between skip-gram trained with negative sampling ( SGNS ) and traditional low-rank approximation methods ( Kolda & Bader , 2009 ; Anandkumar et al. , 2014 ) , showing the equivalence of SGNS optimization to factorizing a shifted PMI matrix ( Church & Hanks , 1990 ) . This equivalence was later retrieved from diverse assumptions ( Assylbekov & Takhanov , 2019 ; Allen et al. , 2019 ; Melamud & Goldberger , 2017 ; Arora et al. , 2016 ; Li et al. , 2015 ) , and exploited to compute closed form expressions approximated in different graph embedding models ( Qiu et al. , 2018 ) . In this work , we refer to the shifted PMI matrix also as SPMIκ = PMI− log κ , where κ is the number of negative samples . Random walk based graph embeddings . Given an undirected , weighted and connected graph G = ( V , E ) with nodes i , j ∈ V , edges ( i , j ) ∈ E and adjacency matrix A , graph embedding methods are unsupervised models designed to map nodes into dense d-dimensional representations ( d |V| ) ( Hamilton et al. , 2017 ) . A well known family of approaches based on the skip-gram model consists in sampling random walks from the graph and processing node sequences as textual sentences . In DEEPWALK ( Perozzi et al. , 2014 ) and NODE2VEC ( Grover & Leskovec , 2016 ) , the skip-gram model is used to obtain node embeddings from co-occurrences in random walk realizations . Although the original implementation of DEEPWALK uses hierarchical softmax to compute embeddings , we will refer to the SGNS formulation given by Qiu et al . ( 2018 ) . Since SGNS can be interpreted as a factorization of the word-context PMI matrix ( Levy & Goldberg , 2014 ) , the asymptotic form of the PMI matrix implicitly decomposed in DEEPWALK can be derived ( Qiu et al. , 2018 ) . Given the 1-step transition matrix P = D−1A , where D = diag ( d1 , . . . , d|V| ) and di = ∑ j∈V Aij is the ( weighted ) node degree , the expected PMI for a node-context pair ( i , j ) occurring in a T -sized window is : E [ PMIDEEPWALK ( i , j ) | T ] = log ( 1 2T ∑T r=1 [ p ∗ ( i ) ( Pr ) ij + p ∗ ( j ) ( Pr ) ji ] p∗ ( i ) p∗ ( j ) ) ( 2.1 ) where p∗ ( i ) = divol ( G ) is the unique stationary distribution for random walks ( Masuda et al. , 2017 ) and vol ( G ) = ∑i , j∈V Aij . We will use this expression in Section 3.2 to build PMI tensors from higher-order graph representations . Time-varying graphs and their algebraic representations . Time-varying graphs ( Holme & Saramäki , 2012 ) are defined as triples H = ( V , E , T ) , i.e . collections of events ( i , j , k ) ∈ E , representing undirected pairwise relations among nodes at discrete times ( i , j ∈ V , k ∈ T ) . H can be seen as a temporal sequence of static graphs { G ( k ) } k∈T , each of those with adjacency matrix A ( k ) such that A ( k ) ij = ω ( i , j , k ) ∈ R is the weight of the event ( i , j , k ) ∈ E . We can concatenate the list of time-stamped snapshots [ A ( 1 ) , . . . , A ( |T | ) ] to obtain a single 3rd-order tensor Astat ( H ) ∈ R|V|×|V|×|T | which characterize the evolution of the graph over time . This representation has been used to discover latent community structures of temporal graphs ( Gauvin et al. , 2014 ) and to perform temporal link prediction ( Dunlavy et al. , 2011 ) . Indeed , beyond the above stacked graph representation , more exhaustive representations are possible . In particular , the multi-layer approach ( De Domenico et al. , 2013 ) allows to map the topology of a time-varying graph H into a static network GH = ( VH , EH ) ( the supra-adjacency graph ) such that vertices of GH correspond to pairs ( i , k ) ≡ i ( k ) ∈ V × T of the original time-dependent network . This representation can be stored in a 4th-order tensor Adyn ( H ) ∈ R|V|×|V|×|T |×|T | equivalent , up to an opportune reshaping , to the adjacency matrix A ( GH ) ∈ R|V||T |×|V||T | associated to GH . Multi-layer representations for time-varying networks have been used to study time-dependent centrality measures ( Taylor et al. , 2019 ) and properties of spreading processes ( Valdano et al. , 2015 ) . Time-varying graph representation learning . Given a time-varying graph H = ( V , E , T ) , we define as temporal network embedding a model that learns from data , implicitly or explicitly , a mapping function : f : ( v , t ) ∈ V × T 7→ v ( t ) ∈ Rd ( 2.2 ) which project time-stamped nodes into a latent low-rank vector space that encodes structural and temporal properties of the original evolving graph . Many existing methods learn node representations from sequences of static snapshots through incremental updates in a streaming scenario : deep autoencoders ( Goyal et al. , 2017 ) , SVD ( Zhang et al. , 2018 ) , skip-gram ( Du et al. , 2018 ) and random walk sampling ( Béres et al. , 2019 ; Mahdavi et al. , 2018 ; Yu et al. , 2018 ) . Another class of models learn dynamic node representations by recurrent/attention mechanisms ( Goyal et al. , 2020 ; Li et al. , 2018 ; Sankar et al. , 2020 ; Xu et al. , 2020 ) or by imposing temporal stability among adjacent time intervals ( Zhou et al. , 2018 ; Zhu et al. , 2016 ) . DYANE ( Sato et al. , 2019 ) and WEG2VEC ( Torricelli et al. , 2020 ) project the dynamic graph structure into a static graph , in order to compute embeddings with WORD2VEC . Closely related to these ones are Nguyen et al . ( 2018 ) and Zhan et al . ( 2020 ) , which learn node vectors according to time-respecting random walks or spreading trajectory paths . Moreover , Kumar et al . ( 2019 ) proposed an embedding framework for user-item temporal interactions , and Malik et al . ( 2020 ) suggested a tensor-based convolutional architecture for dynamic graphs . Methods that perform well for predicting outcomes of spreading processes make use of timerespecting supra-adjacency representations such as the one proposed by Valdano et al . ( 2015 ) . In this representation , random itineraries correspond to temporal paths of the original time-varying graph . The supra-adjacency representation GH that we refer in Section 3.2 , also used in DYANE , with adjacency matrix A ( GH ) , is defined by two rules : 1 . For each event ( i , j , t0 ) , if i is also active at time t1 > t0 and in no other time-stamp between the two , we add a cross-coupling edge between supra-adjacency nodes j ( t0 ) and i ( t1 ) . In addition , if the next interaction of j with other nodes happens at t2 > t0 , we add an edge between i ( t0 ) and j ( t2 ) . The weights of such edges are set to ω ( i , j , t0 ) . 2 . For every case as described above , we also add self-coupling edges ( i ( t0 ) , i ( t1 ) ) and ( j ( t0 ) , j ( t2 ) ) , with weights set to 1 . Figure 1 shows the differences between a time-varying graph and its time-aware supra-adjacency representation , according to the formulation described above . DYANE computes , given a node i ∈ V , one vector representation for each time-stamped node i ( t ) ∈ V ( T ) = { ( i , t ) ∈ V×T : ∃ ( i , j , t ) ∈ E } of this supra-adjacency representation . Similar models that learn time-resolved node representations require a quantity O ( |V| · |T | ) of embedding parameters to represent the system in the latent space . Compared with these methods , our approach requires a quantity O ( |V|+ |T | ) of embedding parameters for disentangled node and time representations .

Summary: In this paper, the authors studied the problem of time-varying graph embedding problems. The authors generalized skip-gram based graph embedding method to time-varying graphs. The authors show that the method can be used to factorize time-varying graphs as high-order tensors via negative sampling. The authors carried out experiments on several time-resolved proximity networks with comparison to several state-of-art baselines.