Scalable Hierarchical Embeddings of Complex Networks

Graph representation learning has become important in order to understand and predict intrinsic structures in complex networks . A variety of embedding methods has in recent years been developed including the Latent Distance Modeling ( LDM ) approach . A major challenge is scaling network embedding approaches to very large networks and a drawback of LDM is the computational cost invoked evaluating the full likelihood having O ( N2 ) complexity , making such analysis of large networks infeasible . We propose a novel multiscale hierarchical estimate of the full likelihood of LDMs providing high-details where the likelihood approximation is most important while scaling in complexity at O ( N logN ) . The approach relies on a clustering procedure approximating the Euclidean norm of every node pair according to the multiscale hierarchical structure imposed . We demonstrate the accuracy of our approximation and for the first time embed very large networks in the order of a million nodes using LDM and contrast the predictive performance to prominent scalable graph embedding approaches . We find that our approach significantly outperforms these existing scalable approaches in the ability to perform link prediction , node clustering and classification utilizing a surprisingly low embedding dimensionality of two to three dimensions whereas the extracted hierarchical structure facilitates network visualization and interpretation . The developed scalable hierarchical embedding approach enables accurate low dimensional representations of very large networks providing detailed visualizations that can further our understanding of their properties and structure . 1 INTRODUCTION . Networks naturally arise in a plethora of scientific areas to model the interactions between entities from physics to sociology and biology , with many instances such as collaboration , protein-protein interaction , and brain connectivity networks ( Newman , 2003 ) . In recent years Graph Representation Learning ( GRL ) approaches have attracted great interest with their outstanding performance compared to the classical techniques for the challenging network analysis problems such as link prediction ( Liben-Nowell & Kleinberg , 2003 ; Backstrom & Leskovec , 2011 ) , node classification ( Getoor & Taskar , 2007 ; Grover & Leskovec , 2016 ) , and community detection ( Fortunato , 2010 ) . Many existing GRL methods ( Hamilton et al. , 2017b ; Zhang et al. , 2020 ) mainly aim to capture the underlying intrinsic relationships among the nodes by either performing random walks ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) over the network or designing a matrix capturing the first and high order node proximities ( Cao et al. , 2015 ; Ou et al. , 2016 ) . However , they require high computational and space costs because of the exact node sampling procedures or the expensive factorization of dense proximity matrices . The recent Graph Neural Networks ( GNNs ) ( Hamilton et al. , 2017b ; Zhang et al. , 2020 ; Wang et al. , 2016 ) methods provide effective tools in learning the node representations by leveraging the side information such as node attribute features ; nevertheless , they also face computational difficulties , especially for large-scale networks consisting of millions of nodes and edges . Although the recent studies aim to alleviate the computational burden of the algorithms through matrix sparsification tools ( Qiu et al. , 2019 ) or hierarchical representations ( Bhowmick et al. , 2020 ; Chen et al. , 2018 ) , the performance of the methods in the downstream tasks significantly drops , and they require larger embedding sizes to compensate for the loss . Latent Space Models ( LSMs ) for the representation of graphs have been quite popular over the past years , especially for social networks analysis ( Hoff et al. , 2002 ) . LSMs utilize the generalized linear model framework to obtain informative latent node embeddings while preserving network characteristics . The choice of latent effects in modeling the link probabilities between the nodes leads to different expressive capabilities characterizing network structure . We consider the Latent Distance Model ( LDM ) ( Hoff et al. , 2002 ) with the Euclidean norm , in which nodes are placed closer in the latent space if they are similar or vice-versa . LDM obeys the triangle inequality and thus naturally represents transitivity and network homophily . These methods are attractive due to their simplicity , as they define well-structured inference problems and are characterized by high explanatory power . The time and space complexities are their main drawbacks , which scale quadratically with the number of nodes in the graph . Many real-world networks can be expressed as hierarchical structures of different scales ( Ravasz & Barabási , 2003 ) . For this purpose , several hierarchical network modeling tools have been proposed , such as the extensions of the stochastic block model to binary and multifurcating hierarchical structures ( Clauset et al. , 2008 ; Roy et al. , 2007 ; Blundell et al. , 2012 ; Herlau et al. , 2012 ; 2013 ) as well as agglomerative ( Blondel et al. , 2008 ; Ahn et al. , 2010 ) and recursive partitioning procedures ( Li et al. , 2020 ) relying on various measures of similarity . Learning the node representations preserving the hierarchical structure of the network is also a very promising task , and it can facilitate the visualization and the understanding of the inner dynamics of the network . In this work , we propose the Scalable Hierarchical Latent Distance Model ( SH-LDM ) combining embedding and hierarchical representations for graph representation learning . Importantly , the hierarchical structure imposed in ( SH-LDM ) reduces the total time and space complexity of the LDM to linearithmic in terms of the number of nodes ( i.e. , O ( N logN ) ) at the same time providing accurate interpretable representation of structure at different scales . Using the SH-LDM we embed moderate sized and large-scale networks containing more than a million nodes and establish the performance of LDM in terms of link prediction and node classification to existing prominent scalable graph embedding approaches . We further highlight how the inferred hierarchical organization can facilitate accurate visualization of network structure even when using only D = 2 dimensional representations providing favorable performance in all the considered GRL tasks ; link-prediction , node classification , node clustering , and network reconstruction . In summary , our contributions are to reconcile embedding and hierarchical representations providing accurate linearithmic approximation of the full likelihood , efficient inference , enhanced visualization and network compression utilizing ultra-low embedding dimensions and hierarchical representations . 2 THE SCALABLE HIERARCHICAL-LATENT DISTANCE MODEL . We presently concentrate our study on the case of undirected networks , but we note that our approach generalizes to both directed and bipartite graphs as described in the supplementary material . Let G = ( V , E ) be a graph where N : = |V | is the number of nodes and YN×N = [ yi , j ] be the adjacency matrix of the graph such that yi , j = 1 if there is an edge between the nodes vi and vj and otherwise it is equal 0 for all 1 ≤ i < j ≤ N . A Latent Space Model ( LSM ) defines a RD-dimensional latent space in which every node of the graph is characterized through the unobserved but informative node-specific variables { zi ∈ RD } . These variables are considered sufficient to describe and explain the underlying relationships between the nodes of the network , such as transitivity and homophily . The probability of an occurring edge between an ordered pair of the graph is considered conditionally independent given the unobserved latent positions . Consequently , the total probability distribution of the network can be written as : P ( Y |Z , θ ) = N∏ i < j p ( yi , j |zi , zj ) , ( 1 ) A popular and convenient parameterization of equation 1 for binary data is through the logistic regression model ( Hoff et al. , 2002 ; Handcock et al. , 2007 ; Krivitsky et al. , 2009 ; Hoff , 2005 ) . In contrast , we adopt the Poisson regression model as proposed in Hoff ( 2005 ) under a generalized linear model framework for the LSM . The use of a Poisson likelihood for modelling binary relationships in a network does not decrease the predictive performance nor the ability of the model to detect the network structure , as shown in Wind & Mørup ( 2012 ) and also generalize the analysis to integer weighted graphs . In addition , the exchange of the logit to a log link function when transitioning from a Bernoulli to a Poisson model defines nice decoupling properties over the predictor variables in the likelihood ( Karrer & Newman , 2011 ; Herlau et al. , 2014 ) . Utilizing the Poisson Latent Distance Model ( LDM ) of the LSM family framework , the rate of an occurring edge depends on a distance metric between the latent positions of the two nodes . We consider the LDM with node-specific biases or random-effects ( Hoff , 2005 ; Krivitsky et al. , 2009 ) such that the expression for the Poisson rate becomes : λij = exp ( γi + γj − d ( zi , zj ) ) . ( 2 ) where γi denotes the node-specific random-effects and dij ( · , · ) denotes any distance metric obeying the triangle inequality { dij ≤ dik + dkj , ∀ ( i , j , k ) } . Considering variables z as the latent charac- teristics , Eq . equation 2 shows that similar nodes will be placed closer in the latent space , yielding a high probability of an occurring edge and thus modeling homophily and satisfies network transitivity and reciprocity through the triangle inequality whereas the node specific bias can account for degree heterogeneity . The conventional LDM utilizing a global bias , γg , corresponds to the special case in which γi = γj = 0.5γg . As in Hoff et al . ( 2002 ) , we presently adopt the Euclidean distance as the choice for the distance metric dij ( · , · ) . 2.1 SCALING THE LATENT DISTANCE MODEL . Optimizing the LDM requires the computation of the log-likelihood which is defined as the sum over each ordered pair of the network as : logP ( Y |λ ) = ∑ i < j ( yij log ( λij ) − λij ) = ∑ i < j : yij=1 log ( λij ) − ∑ i < j λij , ( 3 ) For brevity , we presently ignore the linear scaling of the above log-likelihood by dimensionality D. Large networks are highly sparse ( Barabási & Pósfai , 2016 ) with the number of edges for very sparse networks being proportional to the number of nodes in the network . As a result , the computation of the link contribution ∑ yi , j=1 log ( λi , j ) is relatively cheap scaling linearithmic or sub-linearithmic ( see also supplementary material ) . This is not the case for the second term which still requires the computation of all node pairs scaling as O ( N2 ) making the evaluation of the above likelihood infeasible for large networks . To reduce the complexity , we propose to approximate the non-link term O ( N2 ) using blocks , i.e. , akin to stochastic block models White et al . ( 1976 ) ; Holland et al . ( 1983 ) ; Nowicki & Snijders ( 2001 ) , in which we when grouping the nodes into K clusters define the rate between block k and k′ in terms of their distance between centroids , ∑ i < j λij ≈ K∑ k ( ∑ i , j∈Ck e ( γi+γj−||zi−zj ||2 ) + ∑ i∈k , j /∈Ck e ( γi+γj−||µk−µk′ ||2 ) ) = ( K∑ k ∑ i , j∈Ck e ( γi+γj−||zi−zj ||2 ) ) + ∑ k > k′ e−||µk−µk′ ||2 ( ∑ i∈Ck eγi ) ( ∑ j∈Ck′ eγj ) , ( 4 ) where µk denotes the k ’ th cluster centroid of the set C = { C1 , . . . , CK } , and has absorbed the dependency over the variables Z . Overall , considering the main principle of the LDM that connected and homophilic nodes will be placed closer in the latent space , this approximation generalizes this principle introducing a clustering procedure that obeys ” cluster-homophily ” and ” cluster-transitivity ” over the latent space . More specifically , we can assume that closely related nodes will be positioned in the same cluster while related or interconnected clusters will also be positioned close in the latent space , providing an accurate approximation schema . Assuming equally sized clusters having N/K nodes the first part scales O ( N2/K ) whereas the second part scales O ( K2 ) . As such , there is an undesirable inherent trade-off in which the first term reduces by K but the second term increases quadratically . Thus , by setting K = N/log ( N ) we reduce the first part to scale asO ( N logN ) but at the cost of the second term scaling O ( N2/log ( N ) 2 ) which for large networks is still prohibitive .

Summary: The focus of the paper is in suggesting a node embedding method that uses hierarchy to ensure scalability. The proposed method, “Scalable Hierarchical Latent Distance Model” ( SH-LDM), aims to reconcile embedding and hierarchical network representations. This method is based on the following components: -	__Foundational component:__ a Latent Distance Model. Latent Space Models are a special type of node embedding method that uses  the generalized linear model framework to obtain latent node embeddings while preserving network characteristics. In the case of the Latent Distance Models, this means that nodes are placed closer in the latent space if they are closer together. In particular, here, __representing the edges as sampled from a Poisson distribution__ with parameter $\lambda_{ij}$: $$ \lambda_{ij} = \gamma_i + \gamma_j - || z_i -z_j||_2 $$ Finding a solution in terms of $z_i$ s of this model --- which is not new --- is the object of this paper.  - __Component 2:  Hierarchy:__ To make the fitting of this model more scalable, the authors approximate the distance r $ d(z_, z_j) = || z_i  - z_j|| $ by the distance between centroids $|| \mu_{C_i} -\mu_{C_j}||, \quad \forall C_i\neq C_j$.  - __Component 3: Scalable division of the data in clusters__ The clusters are not known. So the authors use a multiresoluiton KD tree to split the data into K clusters of equal sizes. It further relies on an optimization procedure for k-means clustering with Euclidean norm utilizing the auxiliary function framework of Tsutsu & Morikawa to perform optimization scalably. This allows the partition of the data into $K = \log(N)$ clusters.  This results in a reduction of  the total time and space complexity of the LDM to O(N logN)). The authors then proceed to validate their method using a set of experiments:   1. node classification   2. edge prediction   3. clustering and hierarchical structure recovered in the latent space.  with reasonable performance on a subset of the dataset.