An Interpretable Graph Generative Model with Heterophily

Many models for graphs fall under the framework of edge-independent dot product models . These models output the probabilities of edges existing between all pairs of nodes , and the probability of a link between two nodes increases with the dot product of vectors associated with the nodes . Recent work has shown that these models are unable to capture key structures in real-world graphs , particularly heterophilous structures , wherein links occur between dissimilar nodes . We propose the first edge-independent graph generative model that is a ) expressive enough to capture heterophily , b ) produces nonnegative embeddings , which allow link predictions to be interpreted in terms of communities , and c ) optimizes effectively on realworld graphs with gradient descent on a cross-entropy loss . Our theoretical results demonstrate the expressiveness of our model in its ability to exactly reconstruct a graph using a number of clusters that is linear in the maximum degree , along with its ability to capture both heterophily and homophily in the data . Further , our experiments demonstrate the effectiveness of our model for a variety of important application tasks such as multi-label clustering and link prediction . 1 INTRODUCTION . Graphs naturally arise in data from a variety of fields including sociology ( Mason & Verwoerd , 2007 ) , biology ( Scott , 1988 ) , and computer networking ( Bonato , 2004 ) . A key underlying task in machine learning for graph data is forming models of graphs which can predict edges between nodes , form useful representations of nodes , and reveal interpretable structure in the graph , such as detecting clusters of nodes . Many graph models fall under the framework of edge-independent graph generative models , which can output the probabilities of edges existing between any pair of nodes . The parameters of such models can be trained iteratively on the network , or some fraction of the network which is known , in the link prediction task , e.g. , by minimizing a cross-entropy loss . To choose among these models , one must consider whether the model is capable of expressing structures of interest in the graph , as well as the interpretability of the model . Expressiveness As real-world graphs are high-dimensional objects , graph models generally compress information about the graph . Such models are exemplified by the family of dot product models , which associate each node with a real-valued “ embedding ” vector ; the predicted probability of the link between two nodes increases with the dot product of their embedding vectors . These models can alternatively be seen as factorizing the adjacency matrix of the graph in terms of a low-rank matrix . Recent work ( Seshadhri et al. , 2020 ) has shown that dot product models are limited in their ability to model common structures in real-world graphs , such as triangles incident only on low-degree nodes . In response , Chanpuriya et al . ( 2020 ) showed that with the logistic PCA ( LPCA ) model , which has two embeddings per node ( i.e . using the dot product of the “ left ” embedding of one node and the “ right ” embedding of another ) , not only can such structures be represented , but further , any graph can be exactly represented with embedding vectors whose lengths are linear in the maximum degree of the graph . Peysakhovich & Bottou ( 2021 ) show that the limitations of the single-embedding model , which are overcome by having two embeddings , stem from only being able to represent adjacency matrices which are positive semi-definite , which prevents them from representing heterophilous structures in graphs ; heterophilous structures are those wherein dissimilar nodes are linked . Heterophily : Motivating example To demonstrate how heterophily can manifest in networks , as well as how models which assume homophily can fail to represent such networks , we provide a simple synthetic example . Suppose a recruiting website allows its members to contact each other ; we construct a graph of these members , with an edge indicating that two members have been in contact . Members are either recruiters or non-recruiters , and each member comes from one of ten locations ( e.g. , a city ) . Members from the same location are likely to contact each other ; this typifies homophily , wherein links occur between similar nodes . Furthermore , recruiters are unlikely to contact other recruiters , and non-recruiters are unlikely to contact other non-recruiters ; this typifies heterophily . Figure 1 shows an instantiation of such an adjacency matrix with 1000 nodes , which are randomly assigned to one of the ten locations and one of recruiter / non-recruiter . We recreate this network with our embedding model and the BIGCLAM algorithm of Yang & Leskovec ( 2013 ) , which explicitly assumes homophily . We also compare with the best low-rank approximation to the adjacency matrix in terms of Frobenius error ; this is the SVD of the matrix , discarding all but the top singular values . In Figure 1 , we show how BIGCLAM captures only the ten communities based on location , i.e. , only the homophilous structure , and fails to capture the heterophilous distinction between recruiters and non-recruiters . We also plot the error of the reconstructions as the embedding length increases . There are 10 · 2 = 20 different kinds of nodes , meaning the expected adjacency matrix is rank-20 , and our model maintains the lowest error up to this embedding length ; by contrast , BIGCLAM is unable to decrease error after capturing location information with length-10 embeddings . Interpretability Beyond being able to capture a given network accurately , it is often desirable for a graph model to form interpretable representations of nodes and to produce edge probabilities in an interpretable fashion . Dot product models can achieve this by restricting the node embeddings to be nonnegative . Nonnegative factorization has long been used to decompose data into parts ( Donoho & Stodden , 2003 ) . In the context of graphs , this entails decomposing the set of nodes of the network into clusters or communities . In particular , each entry of the nonnegative embedding vector of a node represents the intensity with which the node participates in a community . Note that this allows the edge probabilities output by dot product models to be interpretable in terms of coparticipation in communities . Depending on the model , these vectors may have restrictions such as a sum-to-one requirement , meaning the node is assigned a categorical distribution over communities . The least restrictive and most expressive case is that of soft assignments to overlapping communities , where the entries can vary totally independently . In models for this case , the output of the dot product is often mapped through a nonlinear link function to produce a probability , i.e . to ensure the value lies in [ 0 , 1 ] . This link function ideally also facilitates straightforward interpretation . We propose the first edge-independent graph generative model that is a ) expressive enough to capture heterophily , b ) interpretable in that it produces nonnegative embeddings , and c ) optimizes effectively on real-world graphs with gradient descent on a cross-entropy loss . Summary of main contributions The key contributions of this work are as follows : • We introduce a graph generative model , based on nonnegative matrix factorization , which is able to represent both heterophily and overlapping communities . Our model outputs link probabilities which are interpretable in terms of the communities it detects . • We provide a scheme for initialization of the nonnegative factors using the arbitrary real factors generated by logistic PCA . We show theoretically how a graph which is represented exactly by LPCA can also be represented exactly by our model . • We show theoretically that , with a small number of communities , our model can exactly represent a natural class of graphs which exhibits both heterophily and overlapping communities . • In experiments , we show that our algorithm is competitive on real-world graphs in terms of representing the network , doing link prediction , and producing communities which align with ground-truth . 2 GRAPH GENERATIVE MODEL . Consider the set of undirected , unweighted graphs on n nodes , i.e. , the set of graphs with symmetric adjacency matrices in { 0 , 1 } n×n . We propose an edge-independent , generative model for such graphs . Given a diagonal matrixW ∈ Rk×k and a matrix V ∈ [ 0 , 1 ] n×k , we set the probability of an edge existing between nodes i and j to be the ( i , j ) -th entry of matrix Ã : Ã : = σ ( V > WV ) , ( 1 ) where σ is the logistic function . k represents the number of clusters ; intuitively , if vi ∈ Rk is the i-th row of matrix V , then vi is soft assignment of node i to the k communities . W can be viewed as a cluster affinity matrix . An equivalent alternative formulation is Ãi , j = σ ( viWv > j ) . ( 2 ) Interpretation The edge probabilities output by this model have an intuitive interpretation , and to maximize interpretability , we focus on the case where W is diagonal . Recall that there is a one-to-one-to-one relationship between probability p ∈ [ 0 , 1 ] , odds o = p1−p ∈ [ 0 , ∞ ) , and logit ` = log ( o ) ∈ ( −∞ , +∞ ) . The logit of the link probability between nodes i and j is v > i Wvj , which is a summation of terms vicvjcWcc over all communities c ∈ [ k ] . If the nodes both fully participate in community c , that is , vic = vjc = 1 , then the edge logit is changed by Wcc starting from a baseline of 0 , or equivalently the odds of an edge is multiplied by exp ( Wcc ) starting from a baseline odds of 1 ; if either of the nodes participates only partially in community c , then the change in logit and odds is accordingly prorated . Homophily and heterophily also have a clear interpretaion in this model : homophilous communities are those withWcc > 0 , where two nodes both participating in the community increases the odds of a link , whereas communities withWcc < 0 are heterophilous , and coparticipation decreases the odds of a link . 3 RELATED WORK . Node clustering There is extensive prior work on the node clustering problem ( Schaeffer , 2007 ; Aggarwal & Wang , 2010 ; Nascimento & De Carvalho , 2011 ) , perhaps the most well-known being the normalized cuts algorithm of Shi & Malik ( 2000 ) , which produces a clustering based on the entrywise signs of an eigenvector of the graph Laplacian matrix . However , the clustering algorithms which are most relevant to our work are those based on non-negative matrix factorization ( NMF ) ( Lee & Seung , 1999 ; Berry et al. , 2007 ; Wang & Zhang , 2012 ; Gillis , 2020 ) . One such algorithm is that of Yu et al . ( 2005 ) , which approximately factors a graph ’ s adjacency matrix A ∈ { 0 , 1 } n×n into two positive matrices H and Λ , where H ∈ Rn×k+ is left-stochastic ( i.e . each of its columns sums to 1 ) and Λ ∈ Rk×k+ is diagonal , such that HΛH > ≈ A . Here H represents a soft clustering of the n nodes into k clusters , while the diagonal entries of Λ represent the prevalence of edges within clusters . Note the similarity of the factorization to our model , save for the lack of a nonlinearity . Other NMF approaches include those of Ding et al . ( 2008 ) , Yang et al . ( 2012 ) , Kuang et al . ( 2012 ) , and Kuang et al . ( 2015 ) ( SYMNMF ) . Modeling heterophily Much of the existing work on graph models has an underlying assumption of network homophily ( Newman , 2002 ; Johnson et al. , 2010 ; Noldus & Van Mieghem , 2015 ) . There has been significant recent interest in the limitations of graph neural network ( GNN ) models ( Duvenaud et al. , 2015 ; Li et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) at addressing network heterophily ( Nt & Maehara , 2019 ; Zhu et al. , 2020 ) , as well as proposed solutions ( Pei et al. , 2020 ; Zhu et al. , 2021 ; Yan et al. , 2021 ) , but relatively less work for more fundamental models such as those for clustering . Some existing NMF approaches to clustering do naturally model heterophilous structure in networks . The model of Nourbakhsh et al . ( 2014 ) , for example , is similar to that of Yu et al . ( 2005 ) , but allows the cluster affinity matrix Λ to be non-diagonal ; this allows for inter-cluster edge affinity to exceed intra-cluster edge affinity , so heterophily can arise in this model , though it is not a focus of their work . Further , the model of Miller et al . ( 2009 ) is similar to ours and also allows for heterophily , though it restricts the cluster assignment matrix V to be binary ; additionally , their training algorithm is not based on gradient descent as ours is , and it does not scale to large networks . More recently , Peysakhovich & Bottou ( 2021 ) propose a decomposition of the form A ≈ D + BB > − CC > , where D ∈ Rn×n is diagonal and B , C ∈ Rn×k are low-rank ; the authors discuss how , interestingly , this model separates the homophilous and heterophilous structure into different factors , namelyB andC . However , this work does not pursue a clustering interpretation or investigate setting the factorsB andC to be nonnegative . One stage of our training algorithm uses a similar decomposition , though it includes the nonnegativity constraint ; this is detailed in Section 4 . Overlapping clustering Many models discussed above focus on the single-label clustering task . We are interested in the closely-related but distinct task of multi-label clustering , also known as overlapping community detection ( Xie et al. , 2013 ; Javed et al. , 2018 ) . The BIGCLAM algorithm of Yang & Leskovec ( 2013 ) uses the following generative model for this task : the probability of a link between two nodes i and j is given by 1 − exp ( −fi · fj ) , where fi , fj ∈ Rk+ represent the intensities with which the nodes participate in each of the k communities . This model allows for intersections of communities to be especially dense with edges , which the authors generally observe in real-world networks ; by contrast , they claim that prior state-of-the-art approaches , including ones based on clustering links ( Ahn et al. , 2010 ) and clique detection ( Palla et al. , 2005 ) , as well as a mixed-membership variant ( Airoldi et al. , 2008 ) of the stochastic block model ( Holland et al. , 1983 ) , implicitly assume that intersections are sparse . BIGCLAM assumes strict homophily of the communities , whereas our model allows for both homophily and heterophily . Additionally , unlike in our model , there is no upper bound to the intensities of community participation ( i.e . the entries of each f ) , so it is unclear how to incorporate prior knowledge about community membership in the form of binary labels , as in a semi-supervised situation . The approach of Zhang & Yeung ( 2012 ) is more similar to ours and more amenable to such prior information in that community assignments are bounded ; specifically , the model is similar to those of Yu et al . ( 2005 ) and Nourbakhsh et al . ( 2014 ) , but allows the cluster assignment matrix H to be an arbitrary matrix of probabilities rather left-stochastic . However , unlike our model and BIGCLAM , these models lack a nonlinear linking function ; recent work outside clustering and community detection on graph generative models ( Rendsburg et al. , 2020 ; Chanpuriya et al. , 2020 ) suggests that the addition of a nonlinear linking function , specifically softmax and logistic nonlinearities as in our model , can make matrix factorization-based graph models more expressive . Lastly , a recent approach is the VGRAPH model of Sun et al . ( 2019 ) , which also lacks a final nonlinear linking function , but , interestingly , has an intermediate linking function : the matrix factors ( i.e . the cluster assignment matrices ) themselves are a product of learned embeddings for the nodes and communities , put through a softmax linking function . Their algorithm ultimately determines overlapping communities as in link clustering approaches , and they find that it generally achieves state-of-the-art results in matching ground-truth communities ; as discussed in Section 6.2 , we find that our algorithm ’ s performance on this task compares favorably to VGRAPH .

Summary: The paper proposes an edge-independent graph generative model. Besides the model, the paper also proposes the training process of the model and some theoretical contributions. Unfortunately, the evaluation of the model is based on another problem, clustering. This makes the paper complicate to evaluate. While the main contributions could be important, its evaluation does not guarantee that is able to replicate real networks. Most importantly, given its current results, it must suffer from the degeneracy problem.