Time-varying Graph Representation Learning via Higher-Order Skip-Gram with Negative Sampling

1 INTRODUCTION . A great variety of natural and artificial systems can be represented as networks of elementary structural entities coupled by relations between them . The abstraction of such systems as networks helps us understand , predict and optimize their behaviour ( Newman , 2003 ; Albert & Barabási , 2002 ) . In this sense , node and graph embeddings have been established as standard feature representations in many learning tasks ( Cai et al. , 2018 ; Goyal & Ferrara , 2018 ) . Node embedding methods map nodes into low-dimensional vectors that can be used to solve downstream tasks such as edge prediction , network reconstruction and node classification . Node embeddings have proven successful in achieving low-dimensional encoding of static network structures , but many real-world networks are inherently dynamic ( Holme & Saramäki , 2012 ) . Timeresolved networks are also the support of important dynamical processes , such as epidemic or rumor spreading , cascading failures , consensus formation , etc . ( Barrat et al. , 2008 ) . Time-resolved node embeddings have been shown to yield improved performance for predicting the outcome of dynamical processes over networks , such as information diffusion and disease spreading ( Sato et al. , 2019 ) , providing estimation of infection and contagion risk when used with contact tracing data . Since we expect having more data on proximity networks being used for contact tracing and as proxies for epidemic risk ( Alsdurf et al. , 2020 ) , learning meaningful representations of time-resolved proximity networks can be of extreme importance when facing events such as epidemic outbreaks ( Kapoor et al. , 2020 ; Gao et al. , 2020 ) . The manual and automatic collection of time-resolved proximity graphs for contact tracing purposes presents an opportunity for quick identification of possible infection clusters and infection chains . Even before the COVID-19 pandemic , the use of wearable proximity sensors for collecting time-resolved proximity networks has been largely discussed in the literature and many approaches have been used to describe patterns of activity and community structure , and to study spreading patterns of infectious diseases ( Sapienza et al. , 2015 ; Gauvin et al. , 2014 ; Génois et al. , 2015 ) . Here we propose a representation learning model that performs implicit tensor factorization on different higher-order representations of time-varying graphs . The main contributions are as follows : Given that the skip-gram embedding approach implicitly performs a factorization of the shifted pointwise mutual information matrix ( PMI ) ( Levy & Goldberg , 2014 ) , we generalize it to perform implicit factorization of a shifted PMI tensor . We then define the steps to achieve this factorization using higher-order skip-gram with negative sampling ( HOSGNS ) optimization . We show how to apply 3rd-order and 4th-order SGNS on different higher-order representations of time-varying graphs . Finally , we show that time-varying graph representations learned via HOSGNS outperform stateof-the-art methods when used to solve downstream tasks , even using a fraction of the number of embedding parameters . We report the results of learning embeddings on empirical time-resolved face-to-face proximity data and using such representations as predictors for solving two different tasks : network reconstruction and predicting the outcomes of a SIR spreading process over the time-varying graph . We compare these results with state-of-the art methods for time-varying graph representation learning . 2 PRELIMINARIES AND RELATED WORK . Skip-gram representation learning . The skip-gram model was designed to compute word embeddings in WORD2VEC ( Mikolov et al. , 2013 ) , and afterwards extended to graph node embeddings ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ) . Levy & Goldberg ( 2014 ) established the relation between skip-gram trained with negative sampling ( SGNS ) and traditional low-rank approximation methods ( Kolda & Bader , 2009 ; Anandkumar et al. , 2014 ) , showing the equivalence of SGNS optimization to factorizing a shifted PMI matrix ( Church & Hanks , 1990 ) . This equivalence was later retrieved from diverse assumptions ( Assylbekov & Takhanov , 2019 ; Allen et al. , 2019 ; Melamud & Goldberger , 2017 ; Arora et al. , 2016 ; Li et al. , 2015 ) , and exploited to compute closed form expressions approximated in different graph embedding models ( Qiu et al. , 2018 ) . In this work , we refer to the shifted PMI matrix also as SPMIκ = PMI− log κ , where κ is the number of negative samples . Random walk based graph embeddings . Given an undirected , weighted and connected graph G = ( V , E ) with nodes i , j ∈ V , edges ( i , j ) ∈ E and adjacency matrix A , graph embedding methods are unsupervised models designed to map nodes into dense d-dimensional representations ( d |V| ) ( Hamilton et al. , 2017 ) . A well known family of approaches based on the skip-gram model consists in sampling random walks from the graph and processing node sequences as textual sentences . In DEEPWALK ( Perozzi et al. , 2014 ) and NODE2VEC ( Grover & Leskovec , 2016 ) , the skip-gram model is used to obtain node embeddings from co-occurrences in random walk realizations . Although the original implementation of DEEPWALK uses hierarchical softmax to compute embeddings , we will refer to the SGNS formulation given by Qiu et al . ( 2018 ) . Since SGNS can be interpreted as a factorization of the word-context PMI matrix ( Levy & Goldberg , 2014 ) , the asymptotic form of the PMI matrix implicitly decomposed in DEEPWALK can be derived ( Qiu et al. , 2018 ) . Given the 1-step transition matrix P = D−1A , where D = diag ( d1 , . . . , d|V| ) and di = ∑ j∈V Aij is the ( weighted ) node degree , the expected PMI for a node-context pair ( i , j ) occurring in a T -sized window is : E [ PMIDEEPWALK ( i , j ) | T ] = log ( 1 2T ∑T r=1 [ p ∗ ( i ) ( Pr ) ij + p ∗ ( j ) ( Pr ) ji ] p∗ ( i ) p∗ ( j ) ) ( 2.1 ) where p∗ ( i ) = divol ( G ) is the unique stationary distribution for random walks ( Masuda et al. , 2017 ) and vol ( G ) = ∑i , j∈V Aij . We will use this expression in Section 3.2 to build PMI tensors from higher-order graph representations . Time-varying graphs and their algebraic representations . Time-varying graphs ( Holme & Saramäki , 2012 ) are defined as triples H = ( V , E , T ) , i.e . collections of events ( i , j , k ) ∈ E , representing undirected pairwise relations among nodes at discrete times ( i , j ∈ V , k ∈ T ) . H can be seen as a temporal sequence of static graphs { G ( k ) } k∈T , each of those with adjacency matrix A ( k ) such that A ( k ) ij = ω ( i , j , k ) ∈ R is the weight of the event ( i , j , k ) ∈ E . We can concatenate the list of time-stamped snapshots [ A ( 1 ) , . . . , A ( |T | ) ] to obtain a single 3rd-order tensor Astat ( H ) ∈ R|V|×|V|×|T | which characterize the evolution of the graph over time . This representation has been used to discover latent community structures of temporal graphs ( Gauvin et al. , 2014 ) and to perform temporal link prediction ( Dunlavy et al. , 2011 ) . Indeed , beyond the above stacked graph representation , more exhaustive representations are possible . In particular , the multi-layer approach ( De Domenico et al. , 2013 ) allows to map the topology of a time-varying graph H into a static network GH = ( VH , EH ) ( the supra-adjacency graph ) such that vertices of GH correspond to pairs ( i , k ) ≡ i ( k ) ∈ V × T of the original time-dependent network . This representation can be stored in a 4th-order tensor Adyn ( H ) ∈ R|V|×|V|×|T |×|T | equivalent , up to an opportune reshaping , to the adjacency matrix A ( GH ) ∈ R|V||T |×|V||T | associated to GH . Multi-layer representations for time-varying networks have been used to study time-dependent centrality measures ( Taylor et al. , 2019 ) and properties of spreading processes ( Valdano et al. , 2015 ) . Time-varying graph representation learning . Given a time-varying graph H = ( V , E , T ) , we define as temporal network embedding a model that learns from data , implicitly or explicitly , a mapping function : f : ( v , t ) ∈ V × T 7→ v ( t ) ∈ Rd ( 2.2 ) which project time-stamped nodes into a latent low-rank vector space that encodes structural and temporal properties of the original evolving graph . Many existing methods learn node representations from sequences of static snapshots through incremental updates in a streaming scenario : deep autoencoders ( Goyal et al. , 2017 ) , SVD ( Zhang et al. , 2018 ) , skip-gram ( Du et al. , 2018 ) and random walk sampling ( Béres et al. , 2019 ; Mahdavi et al. , 2018 ; Yu et al. , 2018 ) . Another class of models learn dynamic node representations by recurrent/attention mechanisms ( Goyal et al. , 2020 ; Li et al. , 2018 ; Sankar et al. , 2020 ; Xu et al. , 2020 ) or by imposing temporal stability among adjacent time intervals ( Zhou et al. , 2018 ; Zhu et al. , 2016 ) . DYANE ( Sato et al. , 2019 ) and WEG2VEC ( Torricelli et al. , 2020 ) project the dynamic graph structure into a static graph , in order to compute embeddings with WORD2VEC . Closely related to these ones are Nguyen et al . ( 2018 ) and Zhan et al . ( 2020 ) , which learn node vectors according to time-respecting random walks or spreading trajectory paths . Moreover , Kumar et al . ( 2019 ) proposed an embedding framework for user-item temporal interactions , and Malik et al . ( 2020 ) suggested a tensor-based convolutional architecture for dynamic graphs . Methods that perform well for predicting outcomes of spreading processes make use of timerespecting supra-adjacency representations such as the one proposed by Valdano et al . ( 2015 ) . In this representation , random itineraries correspond to temporal paths of the original time-varying graph . The supra-adjacency representation GH that we refer in Section 3.2 , also used in DYANE , with adjacency matrix A ( GH ) , is defined by two rules : 1 . For each event ( i , j , t0 ) , if i is also active at time t1 > t0 and in no other time-stamp between the two , we add a cross-coupling edge between supra-adjacency nodes j ( t0 ) and i ( t1 ) . In addition , if the next interaction of j with other nodes happens at t2 > t0 , we add an edge between i ( t0 ) and j ( t2 ) . The weights of such edges are set to ω ( i , j , t0 ) . 2 . For every case as described above , we also add self-coupling edges ( i ( t0 ) , i ( t1 ) ) and ( j ( t0 ) , j ( t2 ) ) , with weights set to 1 . Figure 1 shows the differences between a time-varying graph and its time-aware supra-adjacency representation , according to the formulation described above . DYANE computes , given a node i ∈ V , one vector representation for each time-stamped node i ( t ) ∈ V ( T ) = { ( i , t ) ∈ V×T : ∃ ( i , j , t ) ∈ E } of this supra-adjacency representation . Similar models that learn time-resolved node representations require a quantity O ( |V| · |T | ) of embedding parameters to represent the system in the latent space . Compared with these methods , our approach requires a quantity O ( |V|+ |T | ) of embedding parameters for disentangled node and time representations .

Summary: The paper proposes an implicit tensor factorization approach for learning time-varying node representations over dynamic networks.  The core method lifts the well-known skip gram based embedding approach from matrix to higher order tensors to support temporal dimensions. The authors claim that such tensor based treatment allows to disentangle the role of node and time. Negative sampling method ( similar to noise contrastive estimation) is extended the higher order tensor setting and incorporated in the cross entropy objective for training. In the experiments, the authors consider five variants of face-to-face proximity data that contains temporal interactions and focuses on tasks of node classification (predicting outcome of SIR epidemic process) and link prediction (in the form of event reconstruction). The proposed method has been compared against two discrete time graph representation learning model and a recently proposed tensor based method. The authors claim that the provided method shows comparable performance with requirement to train lesser number of parameters. Also, the authors provide qualitative analysis in terms of embedding visualizations and goodness of fit plots.