An Interpretable Graph Generative Model with Heterophily

Many models for graphs fall under the framework of edge-independent dot product models . These models output the probabilities of edges existing between all pairs of nodes , and the probability of a link between two nodes increases with the dot product of vectors associated with the nodes . Recent work has shown that these models are unable to capture key structures in real-world graphs , particularly heterophilous structures , wherein links occur between dissimilar nodes . We propose the first edge-independent graph generative model that is a ) expressive enough to capture heterophily , b ) produces nonnegative embeddings , which allow link predictions to be interpreted in terms of communities , and c ) optimizes effectively on realworld graphs with gradient descent on a cross-entropy loss . Our theoretical results demonstrate the expressiveness of our model in its ability to exactly reconstruct a graph using a number of clusters that is linear in the maximum degree , along with its ability to capture both heterophily and homophily in the data . Further , our experiments demonstrate the effectiveness of our model for a variety of important application tasks such as multi-label clustering and link prediction . 1 INTRODUCTION . Graphs naturally arise in data from a variety of fields including sociology ( Mason & Verwoerd , 2007 ) , biology ( Scott , 1988 ) , and computer networking ( Bonato , 2004 ) . A key underlying task in machine learning for graph data is forming models of graphs which can predict edges between nodes , form useful representations of nodes , and reveal interpretable structure in the graph , such as detecting clusters of nodes . Many graph models fall under the framework of edge-independent graph generative models , which can output the probabilities of edges existing between any pair of nodes . The parameters of such models can be trained iteratively on the network , or some fraction of the network which is known , in the link prediction task , e.g. , by minimizing a cross-entropy loss . To choose among these models , one must consider whether the model is capable of expressing structures of interest in the graph , as well as the interpretability of the model . Expressiveness As real-world graphs are high-dimensional objects , graph models generally compress information about the graph . Such models are exemplified by the family of dot product models , which associate each node with a real-valued “ embedding ” vector ; the predicted probability of the link between two nodes increases with the dot product of their embedding vectors . These models can alternatively be seen as factorizing the adjacency matrix of the graph in terms of a low-rank matrix . Recent work ( Seshadhri et al. , 2020 ) has shown that dot product models are limited in their ability to model common structures in real-world graphs , such as triangles incident only on low-degree nodes . In response , Chanpuriya et al . ( 2020 ) showed that with the logistic PCA ( LPCA ) model , which has two embeddings per node ( i.e . using the dot product of the “ left ” embedding of one node and the “ right ” embedding of another ) , not only can such structures be represented , but further , any graph can be exactly represented with embedding vectors whose lengths are linear in the maximum degree of the graph . Peysakhovich & Bottou ( 2021 ) show that the limitations of the single-embedding model , which are overcome by having two embeddings , stem from only being able to represent adjacency matrices which are positive semi-definite , which prevents them from representing heterophilous structures in graphs ; heterophilous structures are those wherein dissimilar nodes are linked . Heterophily : Motivating example To demonstrate how heterophily can manifest in networks , as well as how models which assume homophily can fail to represent such networks , we provide a simple synthetic example . Suppose a recruiting website allows its members to contact each other ; we construct a graph of these members , with an edge indicating that two members have been in contact . Members are either recruiters or non-recruiters , and each member comes from one of ten locations ( e.g. , a city ) . Members from the same location are likely to contact each other ; this typifies homophily , wherein links occur between similar nodes . Furthermore , recruiters are unlikely to contact other recruiters , and non-recruiters are unlikely to contact other non-recruiters ; this typifies heterophily . Figure 1 shows an instantiation of such an adjacency matrix with 1000 nodes , which are randomly assigned to one of the ten locations and one of recruiter / non-recruiter . We recreate this network with our embedding model and the BIGCLAM algorithm of Yang & Leskovec ( 2013 ) , which explicitly assumes homophily . We also compare with the best low-rank approximation to the adjacency matrix in terms of Frobenius error ; this is the SVD of the matrix , discarding all but the top singular values . In Figure 1 , we show how BIGCLAM captures only the ten communities based on location , i.e. , only the homophilous structure , and fails to capture the heterophilous distinction between recruiters and non-recruiters . We also plot the error of the reconstructions as the embedding length increases . There are 10 · 2 = 20 different kinds of nodes , meaning the expected adjacency matrix is rank-20 , and our model maintains the lowest error up to this embedding length ; by contrast , BIGCLAM is unable to decrease error after capturing location information with length-10 embeddings . Interpretability Beyond being able to capture a given network accurately , it is often desirable for a graph model to form interpretable representations of nodes and to produce edge probabilities in an interpretable fashion . Dot product models can achieve this by restricting the node embeddings to be nonnegative . Nonnegative factorization has long been used to decompose data into parts ( Donoho & Stodden , 2003 ) . In the context of graphs , this entails decomposing the set of nodes of the network into clusters or communities . In particular , each entry of the nonnegative embedding vector of a node represents the intensity with which the node participates in a community . Note that this allows the edge probabilities output by dot product models to be interpretable in terms of coparticipation in communities . Depending on the model , these vectors may have restrictions such as a sum-to-one requirement , meaning the node is assigned a categorical distribution over communities . The least restrictive and most expressive case is that of soft assignments to overlapping communities , where the entries can vary totally independently . In models for this case , the output of the dot product is often mapped through a nonlinear link function to produce a probability , i.e . to ensure the value lies in [ 0 , 1 ] . This link function ideally also facilitates straightforward interpretation . We propose the first edge-independent graph generative model that is a ) expressive enough to capture heterophily , b ) interpretable in that it produces nonnegative embeddings , and c ) optimizes effectively on real-world graphs with gradient descent on a cross-entropy loss . Summary of main contributions The key contributions of this work are as follows : • We introduce a graph generative model , based on nonnegative matrix factorization , which is able to represent both heterophily and overlapping communities . Our model outputs link probabilities which are interpretable in terms of the communities it detects . • We provide a scheme for initialization of the nonnegative factors using the arbitrary real factors generated by logistic PCA . We show theoretically how a graph which is represented exactly by LPCA can also be represented exactly by our model . • We show theoretically that , with a small number of communities , our model can exactly represent a natural class of graphs which exhibits both heterophily and overlapping communities . • In experiments , we show that our algorithm is competitive on real-world graphs in terms of representing the network , doing link prediction , and producing communities which align with ground-truth . 2 GRAPH GENERATIVE MODEL . Consider the set of undirected , unweighted graphs on n nodes , i.e. , the set of graphs with symmetric adjacency matrices in { 0 , 1 } n×n . We propose an edge-independent , generative model for such graphs . Given a diagonal matrixW ∈ Rk×k and a matrix V ∈ [ 0 , 1 ] n×k , we set the probability of an edge existing between nodes i and j to be the ( i , j ) -th entry of matrix Ã : Ã : = σ ( V > WV ) , ( 1 ) where σ is the logistic function . k represents the number of clusters ; intuitively , if vi ∈ Rk is the i-th row of matrix V , then vi is soft assignment of node i to the k communities . W can be viewed as a cluster affinity matrix . An equivalent alternative formulation is Ãi , j = σ ( viWv > j ) . ( 2 ) Interpretation The edge probabilities output by this model have an intuitive interpretation , and to maximize interpretability , we focus on the case where W is diagonal . Recall that there is a one-to-one-to-one relationship between probability p ∈ [ 0 , 1 ] , odds o = p1−p ∈ [ 0 , ∞ ) , and logit ` = log ( o ) ∈ ( −∞ , +∞ ) . The logit of the link probability between nodes i and j is v > i Wvj , which is a summation of terms vicvjcWcc over all communities c ∈ [ k ] . If the nodes both fully participate in community c , that is , vic = vjc = 1 , then the edge logit is changed by Wcc starting from a baseline of 0 , or equivalently the odds of an edge is multiplied by exp ( Wcc ) starting from a baseline odds of 1 ; if either of the nodes participates only partially in community c , then the change in logit and odds is accordingly prorated . Homophily and heterophily also have a clear interpretaion in this model : homophilous communities are those withWcc > 0 , where two nodes both participating in the community increases the odds of a link , whereas communities withWcc < 0 are heterophilous , and coparticipation decreases the odds of a link . 3 RELATED WORK . Node clustering There is extensive prior work on the node clustering problem ( Schaeffer , 2007 ; Aggarwal & Wang , 2010 ; Nascimento & De Carvalho , 2011 ) , perhaps the most well-known being the normalized cuts algorithm of Shi & Malik ( 2000 ) , which produces a clustering based on the entrywise signs of an eigenvector of the graph Laplacian matrix . However , the clustering algorithms which are most relevant to our work are those based on non-negative matrix factorization ( NMF ) ( Lee & Seung , 1999 ; Berry et al. , 2007 ; Wang & Zhang , 2012 ; Gillis , 2020 ) . One such algorithm is that of Yu et al . ( 2005 ) , which approximately factors a graph ’ s adjacency matrix A ∈ { 0 , 1 } n×n into two positive matrices H and Λ , where H ∈ Rn×k+ is left-stochastic ( i.e . each of its columns sums to 1 ) and Λ ∈ Rk×k+ is diagonal , such that HΛH > ≈ A . Here H represents a soft clustering of the n nodes into k clusters , while the diagonal entries of Λ represent the prevalence of edges within clusters . Note the similarity of the factorization to our model , save for the lack of a nonlinearity . Other NMF approaches include those of Ding et al . ( 2008 ) , Yang et al . ( 2012 ) , Kuang et al . ( 2012 ) , and Kuang et al . ( 2015 ) ( SYMNMF ) . Modeling heterophily Much of the existing work on graph models has an underlying assumption of network homophily ( Newman , 2002 ; Johnson et al. , 2010 ; Noldus & Van Mieghem , 2015 ) . There has been significant recent interest in the limitations of graph neural network ( GNN ) models ( Duvenaud et al. , 2015 ; Li et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) at addressing network heterophily ( Nt & Maehara , 2019 ; Zhu et al. , 2020 ) , as well as proposed solutions ( Pei et al. , 2020 ; Zhu et al. , 2021 ; Yan et al. , 2021 ) , but relatively less work for more fundamental models such as those for clustering . Some existing NMF approaches to clustering do naturally model heterophilous structure in networks . The model of Nourbakhsh et al . ( 2014 ) , for example , is similar to that of Yu et al . ( 2005 ) , but allows the cluster affinity matrix Λ to be non-diagonal ; this allows for inter-cluster edge affinity to exceed intra-cluster edge affinity , so heterophily can arise in this model , though it is not a focus of their work . Further , the model of Miller et al . ( 2009 ) is similar to ours and also allows for heterophily , though it restricts the cluster assignment matrix V to be binary ; additionally , their training algorithm is not based on gradient descent as ours is , and it does not scale to large networks . More recently , Peysakhovich & Bottou ( 2021 ) propose a decomposition of the form A ≈ D + BB > − CC > , where D ∈ Rn×n is diagonal and B , C ∈ Rn×k are low-rank ; the authors discuss how , interestingly , this model separates the homophilous and heterophilous structure into different factors , namelyB andC . However , this work does not pursue a clustering interpretation or investigate setting the factorsB andC to be nonnegative . One stage of our training algorithm uses a similar decomposition , though it includes the nonnegativity constraint ; this is detailed in Section 4 . Overlapping clustering Many models discussed above focus on the single-label clustering task . We are interested in the closely-related but distinct task of multi-label clustering , also known as overlapping community detection ( Xie et al. , 2013 ; Javed et al. , 2018 ) . The BIGCLAM algorithm of Yang & Leskovec ( 2013 ) uses the following generative model for this task : the probability of a link between two nodes i and j is given by 1 − exp ( −fi · fj ) , where fi , fj ∈ Rk+ represent the intensities with which the nodes participate in each of the k communities . This model allows for intersections of communities to be especially dense with edges , which the authors generally observe in real-world networks ; by contrast , they claim that prior state-of-the-art approaches , including ones based on clustering links ( Ahn et al. , 2010 ) and clique detection ( Palla et al. , 2005 ) , as well as a mixed-membership variant ( Airoldi et al. , 2008 ) of the stochastic block model ( Holland et al. , 1983 ) , implicitly assume that intersections are sparse . BIGCLAM assumes strict homophily of the communities , whereas our model allows for both homophily and heterophily . Additionally , unlike in our model , there is no upper bound to the intensities of community participation ( i.e . the entries of each f ) , so it is unclear how to incorporate prior knowledge about community membership in the form of binary labels , as in a semi-supervised situation . The approach of Zhang & Yeung ( 2012 ) is more similar to ours and more amenable to such prior information in that community assignments are bounded ; specifically , the model is similar to those of Yu et al . ( 2005 ) and Nourbakhsh et al . ( 2014 ) , but allows the cluster assignment matrix H to be an arbitrary matrix of probabilities rather left-stochastic . However , unlike our model and BIGCLAM , these models lack a nonlinear linking function ; recent work outside clustering and community detection on graph generative models ( Rendsburg et al. , 2020 ; Chanpuriya et al. , 2020 ) suggests that the addition of a nonlinear linking function , specifically softmax and logistic nonlinearities as in our model , can make matrix factorization-based graph models more expressive . Lastly , a recent approach is the VGRAPH model of Sun et al . ( 2019 ) , which also lacks a final nonlinear linking function , but , interestingly , has an intermediate linking function : the matrix factors ( i.e . the cluster assignment matrices ) themselves are a product of learned embeddings for the nodes and communities , put through a softmax linking function . Their algorithm ultimately determines overlapping communities as in link clustering approaches , and they find that it generally achieves state-of-the-art results in matching ground-truth communities ; as discussed in Section 6.2 , we find that our algorithm ’ s performance on this task compares favorably to VGRAPH .

Summary: This article deals with the task of community detection in the case of mixed membership networks, where a node can have multiple comunity assignments at once. It has been notice that in real-world networks, community overlaps tend to be much denser than the rest of the graph, which is a challenge to usual models (such as the stochastic block model).  The authors introduce a new mixed-membership graph model, where the expected adjacency matrix has the form  $$ \bar A = \sigma( V W V^\top), $$  with $\sigma$ the logistic function. In this model, the matrix $V$ represents the soft community assignments, and $W$ the inter-community similarities. The negative entries of $W$ allow this model to represent the heterophily of the network, wherein vertices are less likely to be connected when they belong to the same community.  They provide a learning algorithm to approximate $V$ and $W$ given the adjacency matrix $A$ of a graph, and show some existence results about such approximations.  In a second part, the authors provide several numerical experiments on real-world datasets, studying both the reconstruction of the matrix $\bar A$ and of the community membership matrix $V$.  They show that, in the chosen datasets, their algorithm outperforms or equals several others in the litterature.