Spectral embedding for dynamic networks with stability guarantees

1 Introduction . Consider a dynamic network in which nodes come and go , change abruptly or evolve , alone or in communities , and form connections accordingly . Our goal is to find a vector representation , Ŷ ( t ) i 2 Rd , for every node i and time t , which could be used for a diversity of downstream analyses , such as clustering , time series analysis , classification , model selection and more . This problem is known as dynamic ( or evolutionary ) network ( or graph ) embedding , and a great number of scalable and empirically successful techniques have been put forward , with recent surveys by [ 47 , 51 ] . We consider , among these , a subset about which it is reasonable to try to establish a certain statistical guarantee . The novelty of this paper is not to propose a new procedure . Instead , it is to demonstrate that an existing procedure , unfolded adjacency spectral embedding ( UASE ) [ 17 ] , has two important stability properties . Given a sequence of symmetric adjacency matrices A ( 1 ) , . . . , A ( T ) 2 { 0 , 1 } n⇥n , where A ( t ) ij = 1 if nodes i and j form an edge at time t , UASE computes the rank d matrix factorisation of A : = ( A ( 1 ) | · · · |A ( T ) ) to obtain A ⇡ X̂Ŷ > using the singular value decomposition ( precise details later ) . The matrix Ŷ 2 RnT⇥d contains , as rows , the desired representations Ŷ ( 1 ) 1 , . . . , Ŷ ( 1 ) n , . . . , Ŷ ( T ) 1 , . . . , Ŷ ( T ) n . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The original motivation for UASE was to analyse multilayer ( or multiplex ) graphs under an appropriate extension of the random dot product graph model [ 17 ] . To evaluate UASE and other procedures on the task of dynamic network embedding , we instead consider the dynamic latent position model A ( t ) ij ind⇠ Bernoulli ⇣ f n Z ( t ) i , Z ( t ) j o⌘ , ( 1 ) for 1 i < j n , t 2 [ T ] , where Z ( t ) i 2 Rk represents the unknown position of node i at time t , and f : Rk ⇥ Rk ! [ 0 , 1 ] is a symmetric function . This model is well-established [ 39 , 22 , 15 , 16 , 33 , 23 , 9 , 43 , 11 , 10 ] , with recent reviews by [ 20 ] and [ 45 ] , and inference usually proceeds on the basis of a parametric model for f ( e.g . logistic in the latent position distance [ 39 ] ) and for the dynamics of Z ( t ) i ( e.g . a Markov process [ 39 ] ) . The model also includes the dynamic degree-corrected , mixed-membership and standard stochastic block models as special cases , which were studied in [ 48 , 52 , 14 , 29 , 50 , 49 , 31 , 4 , 32 , 19 ] . To make statistical sense of UASE under this latent position model , we must somehow connect its output Ŷ ( t ) i to Z ( t ) i . To this end we construct a canonical representative of Z ( t ) i , denoted Y ( t ) i , that UASE can be seen to estimate . Although we make some regularity assumptions on f and the Z ( t ) i , they are not modelled in an explicit , parametric way , and UASE can clearly be used in practice without having a specific model in mind . For example , we do not make a Markovian assumption on the evolution of Z ( t ) i and UASE can be used to uncover periodic behaviours . The key purpose of imposing a dynamic latent position model is to allow us to put down certain embedding stability requirements . Using this framework , we can define precisely what we mean by two nodes , i and j , behaving “ similarly ” at times s and t respectively . In such cases , ideally we would have Ŷ ( s ) i ⇡ Ŷ ( t ) j . Two special cases are : to assign the same position , up to noise , to nodes behaving similarly at a given time ( cross-sectional stability ) and a constant position , up to noise , to a single node behaving similarly across different times ( longitudinal stability ) . To achieve both cross-sectional and longitudinal stability is generally elusive . We show that two plausible alternatives , omnibus [ 26 ] and independent embedding of each A ( t ) , alternately exhibit one form of stability and not the other . More generally , we find existing procedures [ 6 , 27 , 40 , 8 , 54 , 7 , 28 , 5 , 4 , 32 , 37 , 19 ] tend to trade one type of stability off against the other , e.g . via user-specified cost functions . As a side-note , it could be observed that omnibus embedding is not being evaluated on a task for which it was designed , since in the theory of [ 26 ] the graphs are identically distributed . Because the technique is so different from the others , we still feel it makes an interesting addition . Our central contribution is to prove that UASE asymptotically provides both longitudinal and crosssectional stability : for two nodes , i and j , behaving similarly at times s and t respectively , we have Ŷ ( s ) i ⇡ Ŷ ( t ) j and , moreover , Ŷ ( s ) i and Ŷ ( t ) j have asymptotically equal error distribution . We emphasise that these properties hold without requiring any sort of global stability — the network could vary wildly over time but certain nodes still stay fixed . In the asymptotic regime considered , we have n ! 1 , but T fixed so that , for example , our results are relevant to the case of two large graphs . The alternative regime where T ! 1 grows but n is fixed is not easily handled by existing theory and to provide constant updates to UASE ( or omnibus embedding ) in a streaming context presents significant computational challenges . The remainder of this article is structured as follows . Section 2 gives a pedagogical example demonstrating the cross-sectional and longitudinal stability of UASE in a two-step dynamic stochastic block model , while highlighting the instability of omnibus and independent spectral embedding . In Section 3 , we prove a central limit theorem for UASE under a dynamic latent position model , and demonstrate that the distribution satisfies both stability conditions . In Section 4 , we review the stability of other dynamic network embedding procedures . Section 5 presents an example of UASE applied to a dynamic network of social interactions in a French primary school , with a dynamic community detection example given in the main text and a further classification example provided in the Appendix . Section 6 concludes . 2 Motivating example . Suppose it is of interest to uncover dynamic community structure , including communities changing , merging or splitting . The dynamic stochastic block model [ 52 , 50 ] provides a simple explicit model for this , in which two nodes connect at a certain point in time with probability only dependent on their current community membership . Suppose that at times 1 and 2 , we have the following inter-community link probability matrices , B ( 1 ) = 0 BBBB @ 0.08 0.02 0.18 0.10 0.02 0.20 0.04 0.10 0.18 0.04 0.02 0.02 0.10 0.10 0.02 0.06 1 CCCCA , B ( 2 ) = 0 BBBB @ 0.16 0.16 0.04 0.10 0.16 0.16 0.04 0.10 0.04 0.04 0.09 0.02 0.10 0.10 0.02 0.06 1 CCCCA . At time 1 there are four communities present , for example , a node of community 1 connects with a node of community 3 with probability 0.18 . At time 2 , this matrix changes so that communities 1 and 2 have merged , community 3 has moved , whereas community 4 is unchanged . For simplicity , in this example , the community membership of each node is fixed across time-steps . We simulate a dynamic network from this model over these two time steps , on n = 1000 nodes , equally divided among the four communities and investigate the results of three embedding techniques : UASE , omnibus , and independent spectral embedding , displayed in Figure 1 . Note that the different techniques produce embeddings of different dimensions ; UASE embeds into d = 4 dimensions , omnibus embedding d̃ = 7 , while independent spectral embedding has d1 = 4 and d2 = 3 . For visualisation , we show the leading two dimensions for each embedding . Given the dynamics described above , we contend that the following properties would be desirable : 1 . Cross-sectional stability : The embeddings for communities 1 and 2 at time 2 are close . 2 . Longitudinal stability : The embeddings for community 4 at times 1 and 2 are close . Figure 1 illustrates that UASE has both properties : the blue and orange point clouds merge at time 2 , and the red point cloud has the same location and shape over the two time points . On the other hand , omnibus embedding only has longitudinal stability ( the blue and orange point clouds don ’ t merge at time 2 ) , whereas independent spectral embedding only has cross-sectional stability ( the red point cloud is at different locations over the two time points ) . 3 Theoretical background and results . We consider a sequence of random graphs distributed according to a dynamic latent position model , in which the latent position sequences ( Z ( t ) i ) t2 [ T ] ( where we use the shorthand [ T ] = { 1 , . . . , T } ) are independent of each other , and identically distributed according to a joint distribution F on ZT , for a bounded subset Z of Rk . We emphasise that for a fixed i the latent positions Z ( t ) i and Z ( t0 ) i may have different distributions — we are imposing exchangeability over the nodes ( invariance to node labelling ) , but not over time . By extending the function f to be zero outside of its domain of definition , we may view it as an element of L2 ( Rk ⇥Rk ) , and consequently may define a compact , self-adjoint operator A on L2 ( Rk ) by setting Ag ( x ) = Z Rk f ( x , y ) g ( y ) dy ( 2 ) for all g 2 L2 ( Rk ) . The operator A has a—possibly infinite—sequence of non-zero eigenvalues 1 2 . . . with corresponding orthonormal eigenfunctions u1 , u2 , . . . 2 L2 ( Rk ) , such that Auj = juj ( for further details , see for example [ 35 ] ) . We shall make the simplifying assumption that A has a finite number of non-zero eigenvalues , in which case f admits a canonical eigendecomposition f ( x , y ) = DX i=1 iui ( x ) ui ( y ) , ( 3 ) assumed to hold everywhere , where after relabelling we may assume that | 1| . . . | D| . Several families of functions satisfy the finite rank assumption D < 1 , such as the multivariate polynomials [ 35 ] . Moreover , several existing statistical models can be written as dynamic latent position network models in which D < 1 , including the dynamic mixed membership , degreecorrected , and standard stochastic block models . To assume D < 1 , more generally , is tantamount to a claim that , for large n , A ( t ) has ‘ low ’ approximate rank : an overwhelming proportion of its eigenvalues are close to zero . Large matrices with low approximate rank are routinely encountered across numerous disciplines , and the study [ 46 ] provides a hypothesis for this “ puzzling ” general observation . However , given a real network , we might reject the hypothesis that f has low rank ( we must then also reject any type of stochastic block model ) , for example on the basis of triangle counts [ 41 ] . In such a setting , we anticipate that UASE is still consistent and stable if d is allowed to grow sufficiently slowly with n and there exist asymptotic results for adjacency spectral embedding , for the single graph case , showing convergence in Wasserstein distance under assumptions on eigenvalue decay which can be related to the smoothness of f [ 24 ] . However , even if we could obtain such results for UASE , they would not be as powerful as those we present here for the finite rank case , where we show uniform consistency with asymptotic Gaussian error . Theorem 1 . The adjacency matrices A ( 1 ) , . . . , A ( T ) are jointly distributed according to a multilayer random dot product graph model . The multilayer random dot product graph model ( MRDPG , see [ 17 ] ) is a multi-graph analogue of the generalised random dot product graph model [ 36 ] , and is characterised by the existence of matrices ⇤ ( t ) 2 Rd⇥dt and random matrices X 2 Rn⇥d and Y ( t ) 2 Rn⇥dt ( generated according to some joint distribution G ) such that A ( t ) ij ind⇠ Bernoulli Xi⇤ ( t ) Y ( t ) > j ( 4 ) for each t 2 [ T ] and i , j 2 [ n ] , where Xi and Y ( t ) j denote the ith and jth rows of X and Y ( t ) respectively . We refer the reader to the supplemental material for full details of the proof of Theorem 1 , but note that given knowledge of the function f and the underlying distribution F we can construct explicit maps ' : RTk ! Rd and 't : Rk ! Rdt , producing row vectors , and matrices ⇤ ( t ) 2 Rd⇥dt such that f ( Z ( t ) i , Z ( t ) j ) = Xi⇤ ( t ) Y ( t ) > j , ( 5 ) where Xi = ' ( Zi ) , Y ( t ) j = 't ( Z ( t ) j ) and Zi = ( Z ( 1 ) i | · · · |Z ( T ) i ) 2 RTk , for each t 2 [ T ] and i , j 2 [ n ] . Consequently , each Gram matrix P ( t ) = f ( Z ( t ) i , Z ( t ) j ) i , j2 [ n ] ( 6 ) admits a factorisation into a product of low-rank matrices , in which the matrix whose rows are the vectors ' ( Zi ) appears as a common factor . The dimensions dt and d are precisely the ranks of the matrices P ( t ) and their concatenation P = ( P ( 1 ) | · · · |P ( T ) ) respectively . In the theory that follows we will assume that the dimension d is known and fixed , whereas in practice we would usually have to estimate it ( for example by using profile likelihood [ 55 ] ) . We can extend our model to incorporate a range of sparsity regimes by scaling the function f by a sparsity factor ⇢n , which we assume is either constant and equal to 1 , or else tends to zero as n grows , corresponding to dense and sparse regimes respectively . When this factor is present , the maps ' and 't are scaled by a factor of ⇢ 1/2 n . Realising our model as an MRDPG allows us to make precise statements about the asymptotic behaviour of the point clouds obtained through UASE . In particular , it is the existence of a common factor in the decomposition of each of the Gram matrices P ( t ) that gives rise to the stability properties previously demonstrated in the embeddings Ŷ ( t ) , as this matrix in a sense acts as an anchor for the individual point clouds . In [ 17 ] UASE is used to obtain two multi-graph embeddings ; the algorithm below retains only one — the ‘ dynamic ’ component . Algorithm 1 Unfolded adjacency spectral embedding for dynamic networks input symmetric adjacency matrices A ( 1 ) , . . . , A ( T ) 2 { 0 , 1 } n⇥n , embedding dimension d 1 : Form the matrix A = ( A ( 1 ) | · · · |A ( T ) ) 2 { 0 , 1 } n⇥Tn ( column concatenation ) 2 : Compute the truncated singular value decomposition UA⌃AV > A of A , where ⌃A contains the d largest singular values of A , and UA , VA the corresponding left and right singular vectors 3 : Compute the right embedding Ŷ = VA⌃ 1/2 A 2 RTn⇥d and create sub-embeddings Ŷ ( t ) 2 Rn⇥d where Ŷ = ( Ŷ ( 1 ) ; . . . ; Ŷ ( T ) ) ( row concatenation ) return node embeddings for each time period Ŷ ( 1 ) , . . . , Ŷ ( T ) Replacing A with P in this construction yields the noise-free embeddings Ỹ ( t ) , whose rows are known to be linear transformations of the vectors 't ( Z ( t ) i ) ( see the supplemental material for further details ) . A desirable property of UASE is that since the matrices A ( t ) follow an MRDPG model there exist known asymptotic distributional results for the embeddings Ŷ ( t ) . In order to ensure the validity of these results , we will make the assumption that the sparsity factor ⇢n satisfies ⇢n = ! ( n 1 logc ( n ) ) for some universal constant c > 1 . In the results to follow , UASE is shown to be consistent and stable in a uniform sense , i.e . the maximum error of any position estimate goes to zero , under the assumption that the average network degree grows polylogarithmically in n. To achieve this for less than logarithmic growth would be impossible , by any algorithm , because it would violate the information-theoretic sparsity limit for perfect community recovery under the stochastic block model [ 1 ] . In practice this means that the embedding will have high variance when the graphs have few edges . Several approaches have been proposed to make single graph embeddings more robust ( e.g . to sparsity or heterogeneous degrees ) , such as based on the regularised Laplacian [ 3 ] or non-backtracking matrices [ 21 ] , but it is an open and interesting question how to extend them to the dynamic graph setting to achieve both cross-sectional and longitudinal stability . The first of our distributional results states that after applying an orthogonal transformation—which leaves the structure of the resulting point cloud intact—the embedded points Ŷ ( t ) i converge in the Euclidean norm to the noise-free embedded points Ỹ ( t ) i as the graph size grows : Proposition 2 . There exists a sequence of orthogonal matrices W̃ 2 O ( d ) such that max i2 { 1 , ... , n } Ŷ ( t ) i W̃ Ỹ ( t ) i = O ⇣ log 1/2 ( n ) ⇢1/2n n1/2 ⌘ ( 7 ) with high probability for each t. The matrices W̃ are unidentifiable in practice , but are defined to be the solution to the one-mode orthogonal Procrustes problem W̃ = argmin Q2O ( d ) kUAQ UPk2F + kVAQ VPk2F , ( 8 ) where k·k2F denotes the Frobenius norm . We emphasise that the matrix W̃ is the same for each embedding , and so similar behaviour in the noise-free embeddings Ỹ ( t ) is captured by UASE . Our second result states that after applying a second orthogonal transformation the above error converges in distribution to a fixed multivariate Gaussian distribution : Proposition 3 . Let ⇣ = ( ⇣1| · · · |⇣T ) ⇠ F , and for z 2 Z define ⌃t ( z ) = 8 < : E⇣ h f ( z , ⇣t ) 1 f ( z , ⇣t ) · ' ( ⇣ ) > ' ( ⇣ ) i if ⇢n = 1 E⇣ h f ( z , ⇣t ) · ' ( ⇣ ) > ' ( ⇣ ) i if ⇢n ! 0. . ( 9 ) Then there exists a deterministic matrix R⇤ 2 Rd⇥d and a sequence of orthogonal matrices W 2 O ( d ) such that , given z 2 Z , for all y 2 Rd and for any fixed i 2 [ n ] and t 2 [ T ] , P ⇣ n1/2 ( Ŷ ( t ) i W̃ Ỹ ( t ) i ) W y | Z ( t ) i = z ⌘ ! y , R⇤⌃t ( z ) R > ⇤ . ( 10 ) As in our previous results , the matrices W and R⇤ can be explicitly constructed given knowledge of the underlying function f and distribution F ( again , we defer full details to the supplemental material ) . Note that as in Proposition 2 , both constructed matrices are common to all embeddings . We can now demonstrate one of the key advantages that UASE holds over other embedding methods , namely that UASE exhibits both cross-sectional and longitudinal stability , in a sense that we shall now define . We say that two space-time positions ( z , t ) and ( z0 , t0 ) are exchangeable if f ( z , ⇣t ) = f ( z0 , ⇣t0 ) with probability one , where ⇣ = ( ⇣1| · · · |⇣T ) ⇠ F , and that the positions are exchangeable up to degree if f ( z , ⇣t ) = ↵f ( z0 , ⇣t0 ) for some ↵ > 0 . Equivalently , ( z , t ) and ( z0 , t0 ) are exchangeable if conditional on Z ( t ) i = z and Z ( t0 ) j = z 0 the ith row of P ( t ) and jth row of P ( t 0 ) are equal with probability one . Definition 4 . Given a generic method for dynamic network embedding , with output denoted ( Ẑ ( t ) i ) i2 [ n ] ; t2 [ T ] , define the following stability properties : 1 . Cross-sectional stability : Given exchangeable ( z , t ) and ( z0 , t ) , Ẑ ( t ) i and Ẑ ( t ) j are asymptoti- cally equal , with identical error distribution , conditional on Z ( t ) i = z and Z ( t ) j = z 0 . 2 . Longitudinal stability : Given exchangeable ( z , t ) and ( z , t0 ) , Ẑ ( t ) i and Ẑ ( t0 ) i are asymptoti- cally equal , with identical error distribution , conditional on Z ( t ) i = Z ( t0 ) i = z . The following result then shows that UASE exhibits both types of stability : Corollary 5 . Conditional on Z ( t ) i = z and Z ( t0 ) j = z 0 , the following properties hold : 1 . If ( z , t ) and ( z0 , t0 ) are exchangeable then Ŷ ( t ) i and Ŷ ( t0 ) j are asymptotically equal , with identical error distribution . 2 . If ( z , t ) and ( z0 , t0 ) are exchangeable up to degree then Ŷ ( t ) i and ↵Ŷ ( t0 ) j are asymptotically equal and , under a sparse regime , their error distributions are equal up to scale , satisfying ⌃t ( z ) = ↵⌃t0 ( z0 ) . We can gain insight into our theoretical results by applying them in the context of common statistical models . Under the dynamic stochastic block model of Section 2 , a pair ( z , t ) and ( z0 , t0 ) are exchangeable if and only if the corresponding rows of B ( t ) and B ( t 0 ) are identical . Proposition 3 then predicts that the point cloud obtained via UASE should decompose into a finite number of Gaussian clusters , as we observe in Figure 1 . Moreover , when combined with Corollary 5 the equality of the fourth rows of B ( 1 ) and B ( 2 ) implies that we should expect identical clusters ( centre and shape ) corresponding to the fourth community at both time points ( indicating longitudinal stability ) and similarly the equality of the first and second rows of B ( 2 ) tells us that the communities should be indistinguishable at the second time point ( indicating cross-sectional stability ) , behaviours we observe in Figure 1 . Under degree-correction [ 18 , 29 ] , Corollary 5 indicates that we should at least observe cross-sectional and longitudinal stability along ‘ rays ’ representing communities , a behaviour that is exhibited in Figure 2 .

Summary: The paper analyses several consistency (stability) properties of the unfolded adjacency embedding (UASE), a spectral embedding method for dynamically evolving graphs. Specifically it is shown that UASE yields cross-sectionally and longitudinally stable embeddings. This is contrasted with other (spectral) embedding methods which generally do not achieve these properties.